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ABSTRACT 


In  today's  dynamic  combat  environment,  the  importance  of  Close  Air 
Support  (CAS)  has  increased  significantly  due  to  a  greater  need  to  avoid  civilian 
casualties  and  fratricide  while  maintaining  effective  fire  support  for  engaged 
friendly  forces.  Situation  awareness  (SA)  is  a  skill  that  is  extremely  important  in 
conducting  CAS  safely  and  effectively,  and  is  frequently  one  of  the  greatest 
deficiencies  of  new  pilots.  As  budgets  shrink  and  throughput  on  current  training 
solutions  remains  very  low,  it  is  difficult  to  provide  new  CAS  aviators  with  the 
number  of  training  repetitions  needed  to  gain  SA  proficiency.  A  tablet  computer- 
based  CAS  part  task  trainer  (PTT)  to  improve  SA  was  developed  to  address  this. 
This  system  provides  the  visual  and  audio  stimuli  of  the  CAS  battlefield  and 
trains  a  new  pilot  to  observe  the  environment,  listen  to  communications, 
comprehend  what  is  happening,  and  make  timely  inputs  based  on  his  or  her 
understanding  of  the  overall  situation.  Throughout  the  training,  the  system 
provides  questions  to  the  student  to  evaluate  the  student's  level  of  SA  and  to 
emphasize  key  SA  elements.  The  lessons  delivered  by  the  prototype  CAS  PTT 
suggest  that  it  will  be  useful  to  teach  CAS  situation  awareness  and  improve  the 
performance  of  new  pilots.  Continued  development  of  a  series  of  low-cost  part 
task  trainers  that  fill  different  training  gaps  could  result  in  significant  improvement 
in  training  future  generations  of  aviators. 
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I.  INTRODUCTION 


A.  RESEARCH  DOMAIN 

Marine  Corps  aviation  differs  from  other  service  aviation  forces  in  that  it  is 
designed  to  operate  as  part  of  an  integrated  combined-arms  task  force  instead  of 
a  separate  maneuver  element  (Department  of  the  Navy  2000,  1-1).  This 
combined-arms  task  force  is  referred  to  as  the  Marine  Air  Ground  Task  Force 
(MAGTF).  As  part  of  this  integrated  force,  aviation  assets  bolster  the  Marine 
Corps  ground  forces’  limited  surface  fire  support  assets.  One  of  the  direct  air 
support  missions  that  MAGTF  aviation  provides  for  ground  forces  is  close  air 
support  (CAS).  CAS  is  the  direct  fire  support  to  ground  forces  and  is  integrated 
with  the  ground  forces’  scheme  of  maneuver: 

Close  air  support  (CAS)  is  air  action  by  fixed-wing  and  rotary-wing 
aircraft  against  hostile  targets  that  are  in  close  proximity  to  friendly 
forces,  and  requires  detailed  integration  of  each  air  mission  with  the 
fire  and  movement  of  those  forces.  (Chairman  of  the  Joint  Chiefs  of 
Staff  (CJCS)  2003,  1-1) 

The  CAS  environment  is  dynamic,  difficult,  and  demanding,  and  it  is  filled 
with  a  great  deal  of  friction  and  confusion. 

CAS  is  one  of  the  core  elements  on  the  Marine  Light  Attack  Helicopter 
(HMLA)  Squadron’s  Mission  Essential  Task  List  (METL),  meaning  it  is  a  primary 
mission  that  the  HMLA  was  designed  to  perform  (Department  of  the  Navy  2014, 
1-7).  HMLA  pilots  must  be  well  trained  to  handle  the  demands  of  CAS  to  execute 
the  HMLA’s  mission  successfully.  With  the  rules  of  engagement  (ROE)  of  recent 
conflicts,  CAS  is  becoming  more  important  to  all  services,  as  the  risk  of  collateral 
damage  from  indirect  fires  has  made  CAS  the  preferred  type  of  fire  support  to 
ground  maneuver  elements.  The  importance  of  CAS  to  the  MAGTF  is 
tremendous  and  the  training  of  young  HMLA  pilots  to  provide  effective  CAS  to 
the  MAGTF  is  of  the  highest  priority.  Hence,  the  initial  training  phase  for  new 
HMLA  pilots  is  a  crucial  in  developing  competent  CAS  proficient  aviators,  and 
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due  to  this,  HMLA  instructor  pilots  are  always  looking  for  advances  to  improve 
early  CAS  training. 

As  warfare  has  evolved,  so  has  the  tools  used  to  conduct  CAS  evolved. 
Early  CAS  was  conducted  by  releasing  ballistically  guided  ordnance  with  a 
controller  on  the  ground  using  a  map  and  radio  guiding  the  pilot  where  to  shoot. 
Today,  CAS  uses  not  only  ballistically  guided  ordnance,  but  also  laser  and  GPS 
guided  weapons  as  well.  The  ground  control  team  now  utilizes  laser  designators, 
laser  range  finders,  GPS,  advanced  optical  sensors,  and  computers  to  acquire 
and  pass  information  and  targets  to  CAS  aircraft.  This  information  is  often  sent 
between  the  aircraft  and  ground  controller  digitally  as  well  as  via  voice  over  radio 
communications.  The  wealth  of  information  available  today  greatly  enhances 
CAS  operations  by  increasing  the  accuracy  of  ordnance  and  reducing  fratricide. 

B.  RESEARCH  PROBLEM 

CAS  has  come  a  long  way  from  the  days  of  a  controller  telling  a  pilot  to 
drop  a  bomb  on  a  smoke  marker  next  to  a  target  to  the  current  tactics  of  a  missile 
following  laser  energy  from  a  designator  to  the  exact  desired  point  of  impact  on  a 
target.  With  the  advancement  in  CAS  techniques,  aircraft  and  training  facilities 
have  improved  too.  Instrumented  ranges  gather  precise  data  on  marksmanship 
accuracy  on  targets  and  simulated  surface  to  air  missiles  (SAM)  called  “smoky 
SAMs”  actually  fire  small  Styrofoam  model  rockets  for  aircraft  to  evade.  Even 
modern  simulators  create  such  a  high  fidelity  environment  with  state  of  the  art 
visuals  that  pilots  feel  as  if  they  are  in  the  aircraft  flying  real  missions. 
Sophisticated  sensors  record  aircraft  positioning  and  all  communications,  and 
that  type  of  information  allows  for  high  quality  playback  during  debriefs.  However, 
CAS  ground  training  for  pilots  has  advanced  little  with  technology.  Pilots  still  get 
the  majority  of  their  CAS  ground  training  by  reading  printed  publications;  another 
alternative  is  a  face-to-face  discussion  on  procedures  with  an  instructor.  If 
computer-based  training  is  offered,  it  is  usually  in  the  form  of  a  PowerPoint  slide 
show. 
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Much  aviation  training  teaches  pilots  to  execute  correct  procedures  such 
as  completing  checklists,  takeoff  and  landing,  operating  aircraft  systems  and 
talking  on  the  radios.  CAS  requires  more  than  just  procedural  knowledge  though. 
A  CAS  pilot  must  understand  what  is  happening  around  them  and  how  to  act  in 
accordance  with  these  events.  To  conduct  CAS  effectively,  the  pilot  must  not 
only  execute  correct  CAS  procedures  and  communications,  but  the  pilot  must 
also  track  friendly  and  enemy  unit  locations  and  when  ordnance  is  being 
employed.  Maintaining  the  constant  awareness  of  all  of  these  items  places  a 
heavy  load  on  working  memory  and  attention  situation  awareness  (SA)  (Endsley 
2000,  12).  The  situation  awareness  (SA)  that  a  pilot  must  maintain  is  defined  by 
Endsley  (1995)  as  “the  perception  of  the  elements  in  the  environment  within  a 
volume  of  time  and  space,  the  comprehension  of  their  meaning,  and  the 
projection  of  their  status  in  the  near  future”  (65). 

Typical  space  requirements  in  the  CAS  environment  are  items  like  unit 
positions  and  locations  where  rounds  impact.  While  know  when  to  shoot  is  a  time 
requirement.  A  pilot  must  comprehend  what  these  mean,  and  use  that  knowledge 
to  project  what  will  happen;  both  actions  show  just  how  important  SA  is  to  CAS. 
There  has  been  a  concerted  effort  to  make  technological  improvements  to  CAS 
procedures  and  systems  that  improve  CAS  participant’s  SA  such  as  precise 
target  marking  and  digitally  transmitted  information  flow,  but  little  has  been  done 
to  improve  the  ground  training  for  aircrew  and  to  augment  the  acquisition  of  good 
situation  awareness. 

How  to  acquire  that  better  SA  is  a  question  frequently  asked  and  studied 
in  many  aviation  and  combat  arenas  (Kass,  Herschler,  and  Companion  1991, 
105).  A  survey  of  the  research  that  was  done  on  that  subject  suggests  that 
people  who  perform  with  high  levels  of  SA  are  able  to  match  patterns  recognized 
in  current  situations  and  compare  them  with  those  from  previous  experiences, 
and  make  decisions  from  this  information  (Endsley  2000,  16).  While  much  of  this 
experience  is  gained  through  real-world  situations,  it  can  be  learned  through 
training  in  simulated  environments  as  well  (Kass,  Herschler,  and  Companion 
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1991,  110).  Kass,  Herschler,  and  Companion  conducted  a  study  using  a  tank 
simulation  to  demonstrate  if  seeing  and  recognizing  patterns  during  training 
created  an  experience  that  learners  could  adapt  to  other  similar  situations.  Their 
purpose  was  to  determine  if  the  learner  could  transition  from  learning  procedural 
knowledge  to  a  skill-based  learning  through  pattern  recognition.  The  results  of 
their  study  indicated  that  learners  experiencing  training  focused  on  proper 
battlefield  cues  did  gain  improved  pattern  recognition  ability  and  thus  higher  SA. 

Based  on  the  finding  that  subjects  did  better  in  an  armor  battle  after 
receiving  situation  awareness  training  using  a  simulated  battlefield,  it  is  very 
likely  that  pilots  will  receive  similar  benefit  from  a  simulated  CAS  scenario  that 
provides  environment  stimulating  situation  awareness  (Kass,  Herschler,  and 
Companion  1991,  110).  If  pilots  can  connect  simulated  environmental  features  to 
real  world  ones  and  create  long  term  memory  models  that  reduce  working 
memory  requirements  it  could  possibly  lead  to  improved  performance  in  the 
aircraft  during  actual  CAS  missions  (Endsley  2000,  1 3).  Bundling  this  training  into 
a  part  task  trainer  may  give  the  pilot  a  new  tool  that  improves  their  SA. 

C.  RESEARCH  QUESTIONS 

The  following  research  questions  have  been  identified  in  support  of  the 
research  effort  described  in  this  thesis: 

•  What  is  the  feasibility  of  developing  a  tablet  or  laptop  based  virtual 
environment  (VE)  part  task  trainer  (PTT)  aimed  at  improving  new 
HMLA  pilot’s  situation  awareness  for  close  air  support  (CAS) 
missions  in  the  aircraft? 

•  What  is  the  feasibility  of  using  immersive  VE  technology  is  support 
of  training  of  situation  awareness? 

•  What  is  the  overall  user  experience  and  usability  (effectiveness, 
efficiency  and  user  satisfaction)  of  the  final  user  interface? 

D.  SCOPE 

The  scope  of  this  thesis  is  to  expand  upon  research  work  on  a  proof  of 
concept  helicopter  CAS  part  task  trainer  (Attig  2016)  into  a  prototype  that  not 
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only  supports  training  of  procedures  and  communications  (like  the  original 
prototype)  but  it  also  adds  the  concerns  of  situation  awareness  training  during 
CAS  missions  for  junior  HMLA  copilots.  The  resulting  PTT  simulates  the  CAS 
battlefield  environment  and  pilots  are  able  to  gather  situation  information  from  the 
scenario  presented  to  them  in  simulation.  The  subject  pilots  will  then  be  tested  on 
his  (her)  ability  to  correctly  understand  the  situation.  The  capabilities  of  the  final 
PTT  interface  will  be  also  tested  in  a  formal  usability  study;  this  will  allow  us  to 
test  select  elements  of  trainees’  experience  and  gain  more  information  about 
their  ability  to  execute  scenario  presented  in  PTT  and  their  satisfaction  with  the 
interface. 

E.  APPROACH 

This  research  effort  expands  on  the  present  HMLA  CAS  PTT  by  analyzing 
current  HMLA  instructor  pilots’  situation-awareness  training  techniques  and 
implementing  those  techniques  into  the  PTT’s  simulated  virtual  environment.  This 
study  expands  the  CAS  mission  task  analysis  by  Major  Jesse  Attig  (Attig  2016) 
adding  details  relevant  to  situation  awareness  for  all  steps  before  and  during  the 
attack.  Additionally,  the  study  analyzes  the  users’  ability  to  process  audio 
information  for  details  about  his  or  her  environment  and  then  answer  questions  to 
demonstrate  how  well  the  user’s  perception  mirrors  reality. 

Also,  this  study  attempts  to  create  a  user  interface  that  allows  intuitive  and 
expedient  input  of  all  information  required  to  create  an  effective  CAS  training 
scenario.  A  feasibility  study  was  designed  to  determine  if  the  incorporated  visual 
and  auditory  cues  needed  to  support  situation  awareness  are  included  in  the 
PTT.  The  results  of  this  study  will  be  used  to  create  the  structure  for  a  future 
study  using  CAS  trained  pilots  to  provide  feedback  on  system  usability. 

F.  THESIS  STRUCTURE 

The  remaining  chapters  are  summarized  as  follows: 
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Chapter  II  reviews  and  discusses  current  CAS  ground  training  and  the 
current  capabilities  of  the  proof  of  concept  HMLA  CAS  part  task  trainer. 

Chapter  III  discusses  current  situation  awareness  training  and 
measurement  techniques. 

Chapter  IV  provides  a  detailed  task  analysis  on  the  use  of  the  PTT  to 
generate  and  execute  a  CAS  scenario  for  SA  training. 

Chapter  V  details  the  development  of  the  training  system  situation 
awareness  elements,  the  methods  used  to  measure  those  skills,  the  user 
interface  improvement  development,  and  the  design  of  training  scenario  inputs 
and  audio  enhancement. 

Chapter  VI  provides  details  of  the  feasibility  study,  including  the  test  of 
system  performance  and  usability  study  conducted  with  domain  users. 

Chapter  VII  presents  the  results  and  conclusions  of  the  work,  and  lays  out 
the  most  promising  avenues  for  the  future  work. 
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II.  BACKGROUND 


A.  INTRODUCTION 

New  HMLA  pilots  show  up  to  an  operational  fleet  squadron  after  they  have 
been  qualified  to  fly  their  aircraft  and  operate  the  aircraft  systems.  However,  at 
that  point  they  have  received  little  close  air  support  training.  The  new  pilots  are 
familiar  with  a  non-tactical  flight  environment  that  except  for  ordnance  delivery, 
as  well  as  some  tactical  maneuvering  that  is  not  much  different  from  a  civilian 
flying  environment.  Within  six  months,  these  pilots  must  become  combat  ready 
(CR)  copilot.  One  of  the  areas  they  must  become  proficient  in  is  CAS. 

Attaining  combat  readiness  in  CAS  requires  extensive  knowledge  of 
procedures  and  communications.  Pilots  must  also  become  familiar  with  the 
environment  and  be  able  to  perceive  and  understand  the  CAS  battlespace 
environment,  and  be  ready  to  act  in  it.  Currently,  CAS  training  for  pilots  begins 
with  ground  training.  That  training  segment  consists  of  readings  in  standardized 
publications,  followed  by  instructor  pilot  (IP)  led  discussion  sessions,  and 
culminating  in  walk-through  exercises.  This  will  be  the  only  CAS  training  before 
the  pilot  steps  into  the  simulator  for  the  first  syllabus  CAS  event. 

B.  PROBLEM  SPACE 

After  every  CAS  training  mission,  the  HMLA  aircrew  that  just  flew  the 
mission  will  meet  and  debrief  the  flight  in  ready  room.  In  this  debrief,  the  situation 
awareness  will  more  than  likely  be  one  of  the  topics  of  discussion,  especially  if 
one  of  the  pilots  was  a  junior  aviator  flying  one  of  the  H-1  helicopter’s  syllabus 
initial  CAS  training  events.  Many  of  the  debrief  items  concerning  the  flight  will 
probably  be  about  the  lack  of  SA  or  the  loss  of  SA  during  the  mission.  There  is 
good  reason  for  this  issue  to  be  emphasized:  previous  studies  have  shown  that 
poor  or  incorrect  SA  were  found  to  be  a  leading  cause  of  military  aviation 
mishaps  (Endsley  and  Garland  2000,  2-357)  and  a  frequent  cause  of  fratricide  in 
CAS  (Rafferty,  Stanton,  and  Walker  2012,  24). 
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The  doctrinal  CAS  manual,  Department  of  Defense  Joint  Publication  3- 
09.3  lists  “thoroughly  trained  personnel  with  well-developed  skills”  at  the  top  of  its 
list  of  conditions  for  effective  CAS  (CJCS  2009,  1-7).  Extensive  training  is  crucial 
because  the  pilots  must  have  many  repetitions  and  gain  considerable  experience 
before  they  are  ready  to  conduct  a  real  world  CAS  mission.  Most  of  this  training 
will  take  place  in  the  aircraft  and  in  full  aircraft  simulators;  this  is  where  pilots  gain 
most  of  their  experience.  However,  as  budgets  shrink  the  flight  hours  are 
becoming  harder  to  get  and  the  simulators  are  becoming  more  crowded.  This 
creates  throughput  issues  reducing  the  capacity  to  train  pilots,  so  accomplishing 
some  of  that  training  prior  to  stepping  in  the  simulator  or  aircraft  is  crucial. 

C.  CURRENT  TRAINING 

A  new  pilot  will  begin  CAS  training  by  reading  assigned  portions  of 
publications  including  the  JP  3-09.3  Close  Air  Support  for  a  broad  view  of  all 
regulations,  procedures  and  purposes  of  CAS,  and  the  NTTP  3-22.3-UH1  for 
HMLA  specific  CAS  techniques,  tactics  and  procedures.  This  will  be  followed  by 
series  of  five  CAS  lectures  and  two  “chalk  talks”  (United  States  Marine  Corps 
2011).  All  of  these  events  are  to  be  led  by  an  instructor  pilot.  The  lectures  will  be 
in  typical  lecture  format  with  the  focus  on  procedures  and  techniques.  The  “chalk 
talks”  will  be  interactive  with  an  IP  leading  the  discussion. 

The  first  “chalk  talk”  is  titled  “CAS  discussion  and  walkthrough  demo”  and 
the  second  is  “H-1  CAS  TTPs”  (United  States  Marine  Corps  2011).  The  former  is 
first  chance  the  pilot  under  instruction  (PUI)  views  the  dynamic  nature  of  the  CAS 
environment  in  an  interactive  setting.  Instructor  pilots  and  other  CAS  qualified 
pilots  will  play  various  CAS  roles  in  a  walk-through  format.  Usually,  a  large 
drawing  of  the  CAS  objective  area  map  is  laid  out  on  the  ground  in  chalk  and  the 
role  players  are  stationed  at  locations  on  the  map  representing  where  they  would 
be  in  the  real  world.  The  lead  instructor  directs  the  role  players  through  the 
events  of  CAS  scenario  and  drives  the  development  of  the  scenario.  The  role 
players  will  simulate  the  procedures  and  radio  communications  that  would  take 
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place  while  the  IP  explains  the  events  to  the  students.  Each  role  player  will 
verbalize  each  radio  call  and  walk  a  path  across  the  objective  area  map 
representing  the  flight  or  ground  path  of  the  actual  entity  to  build  the  situation  for 
the  PUI. 

The  training  that  Pills  are  exposed  to  focuses  not  only  on  the  CAS 
procedures,  but  also  on  all  events  that  take  place,  as  well  as  events  that  trigger 
follow  on  events.  The  IP  should  draw  the  student’s  attention  to  what 
communications  and  events  trigger  follow  on  events  during  the  course  of  the 
“chalk  talk.”  For  instance,  when  pilot  hears  that  an  aircraft  is  departing  its  holding 
area  to  begin  an  attack,  the  pilot  should  know  where  to  look  in  the  sky  to  see  that 
aircraft  as  it  maneuver — this  is  important  so  that  pilot’s  own  actions  do  not 
interfere  with  that  aircraft.  Also,  based  on  that  departure  time  the  pilot  will  know  if 
that  aircraft  will  meet  its  time  on  target  and  how  that  might  affect  his  follow  on 
attack.  This  is  the  beginning  of  the  development  of  a  pilot’s  situation  awareness 
for  CAS.  This  initial  CAS  SA  ground  training  is  the  only  specified  syllabus  ground 
training  event  that  really  demonstrates  and  teaches  CAS  SA. 

A  new  pilot  may  get  a  higher  number  of  these  CAS  walk-through  “chalk 
talk”  repetitions  if  an  IP  is  available  to  lead  one.  Sometimes  a  group  of  junior 
pilots  will  get  together  and  conduct  their  own  walk  through,  but  they  will  be 
required  to  instruct  themselves  without  any  oversight  to  ensure  that  procedures 
are  correct  and  that  the  training  session  is  beneficial.  Adding  a  PTT  as  an 
additional  training  solution  equipped  with  standardized  scenarios  and  evaluation, 
would  greatly  benefit  the  CAS  ground  training  of  new  HMLA  pilots  and  set  them 
up  to  be  better  prepared  for  their  first  CAS  flight. 

D.  INFLUENCES  ON  DESIGN:  CANDIDATE  SOLUTIONS 

Major  Jesse  Attig  recognized  this  gap  in  the  CAS  training  of  HMLA 
aviators  (Attig  2016).  He  designed  a  proof  of  concept  part  task  trainer  (PTT)  for 
Close  Air  Support  operations;  this  was  a  step  in  the  direction  to  fill  the  gap  in 
CAS  ground  training.  This  trainer  makes  use  of  an  interactive  virtual  environment 
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that  takes  a  pilot  from  launch  to  beginning  of  a  CAS  attack.  The  user  (pilot)  is 
presented  with  a  scenario  and  he  (she)  must  answer  various  preloaded 
questions  that  pop  up  along  a  simulated  flight  route.  The  pilot  is  expected  to 
focus  training  on  the  procedures  and  the  communications  associated  with  CAS. 

Along  with  executing  correct  procedures  and  communications,  an  aviator 
conducting  CAS  must  also  keep  track  of  the  current  location  of  friendly  ground 
and  air  forces,  where  they  are  moving  to,  where  the  targets  are,  when  and  when 
supporting  arms  are  firing,  and  when  and  where  bombs  are  being  dropped.  A 
good  understanding  of  all  those  elements  will  ensure  that  the  pilot  will  be  able  to 
operate  effectively.  Maintaining  a  constant  awareness  of  all  of  these  items  places 
a  heavy  load  on  working  memory  and  attention  (Endsley  2000,  12).  A  pilot  must 
maintain  situation  awareness  including  an  accurate  perception  of  time  and  space 
(Endsley  1995,  65).  For  instance,  those  elements  will  include  space  requirements 
such  as  the  enemy  and  friendly  positions,  positions  where  units  are  firing,  and 
time  requirements  like  when  units  fire  or  drop  ordnance.  A  pilot  must 
comprehend  what  these  mean  and  use  that  to  project  what  will  happen  next 
showing  just  how  important  SA  is  to  CAS. 

How  to  acquire  better  SA  is  a  question  frequently  asked  and  studied 
among  the  researchers  and  practitioners  in  this  domain.  Most  SA  comes  from 
input  to  the  five  senses  (Endsley  2000,  9).  For  CAS  operations,  the  visual  and 
aural  domains  are  the  two  primary  senses  receiving  cues  relevant  to  the  CAS 
mission.  Tactile  and  olfactory  senses  play  a  part,  but  stimulus  to  those  senses  is 
generally  more  relevant  to  aircraft  performance  not  CAS.  Therefore,  the  SA 
training  system  should  provide  visual  and  audio  cues  relevant  to  the  CAS 
environment.  Also,  the  PTT  should  present  information  to  the  user  that  is  similar 
to  the  real  environment. 

The  HMLA  PTT  was  developed  to  help  remedy  the  gap  in  pilot  knowledge 

by  teaching  pilots  CAS  procedures  (Attig  2016).  Procedures  are  the  foundation  of 

effective  CAS,  but  there  is  a  great  deal  of  flexibility  required  in  executing  those 

procedures.  Retired  U.S.  Air  Force  Major  General  I.B.  Holley  states  that  there  is 
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“no  prescription... to  be  followed  slavishly  and  mechanically”  when  conducting 
CAS,  and  those  procedures  “must  sometimes  be  altered  or  adjusted  to  fit  these 
local  variations”  (Cooling  1990,  535).  In  order  to  know  when  a  pilot  must  make 
adjustments,  the  pilot  needs  to  understand  what  is  happening  in  the  environment, 
and  be  able  to  make  decisions  on  how  to  properly  affect  the  outcome.  Endsley 
determined  these  skills  to  be  situation  awareness  (Endsley  1995,  65).  Pilots  gain 
this  skill  through  the  experience  gained  in  training.  In  the  past,  most  of  this 
training  was  accomplished  through  flying  CAS  training  missions  in  the  aircraft. 
However,  with  the  reduction  of  available  flight  hours  looming  ahead  there  must 
be  alternative  mean  to  achieve  proficiency  quickly.  The  goal  for  this  study  is  to 
use  a  virtual  environment  that  simulates  visual  and  audio  sensory  experiences  of 
a  CAS  environment,  and  allow  junior  HMLA  pilots  to  train  CAS  operations  and 
improve  situation  awareness  skills  prior  to  climbing  into  the  aircraft  for  their  first 
mission. 

Attig  surveyed  HMLA  instructor  pilots  (IP)  and  pilots  under  instruction 
(PUI)  on  what  were  the  top  three  CAS  difficulties  for  PUIs,  and  the  results  were 
that  both  groups  listed  situation  awareness  as  the  second  most  important 
difficulty  area  (Attig  2016).  This  consensus  amongst  instructor  and  student 
highlights  the  importance  of  SA  in  CAS  missions.  Situation  awareness  is  also  a 
graded  part  of  the  flight  event  as  well.  By  the  end  of  the  CAS  training  flights,  PUIs 
must  be  able  to  utilize  CAS  procedures  and  communications  correctly  and 
recognize  and  correct  their  own  errors  with  only  occasional  input  from  the  IP. 
This  is  according  to  the  official  grading  criteria  per  the  official  USMC  Aviation 
Tracking  Form  derived  from  the  H-1  Training  and  Readiness  (T&R)  Manuals 
(Department  of  the  Navy  2014,  1-6).  In  the  survey  to  the  IPs,  the  instructors 
were  asked  if  they  though  the  PUIs  were  able  to  recognize  and  correct  mistakes 
without  IP  input.  The  IPs  responded  that  the  majority  of  PUI  could  not  do  this. 

Based  on  these  results  and  potentially  severe  consequences  of  having 
incorrect  SA  during  a  CAS  mission,  it  can  be  concluded  that  it  is  imperative  for 
PUIs  to  receive  additional  training  to  improve  their  SA.  For  many  years,  gaining 
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SA  was  accomplished  only  through  experience  from  performing  the  actual  task 
(Endsley  2000,  3).  Most  of  that  experience  for  HMLA  pilots  comes  from  flying 
CAS  training  missions.  In  today’s  fiscally  strained  environment  it  is  difficult  for  a 
squadron  to  fly  the  amount  of  flight  hours  to  get  pilots  the  experience  required  to 
perform  at  a  high  SA  level.  Increased  ground  training  or  simulator  training  are  the 
only  options  available,  but  even  simulator  time  is  hard  to  come  by  with  so  many 
pilots  needing  trained.  That  leaves  ground  training  as  the  only  viable  option. 
Unfortunately,  the  only  real  method  that  includes  any  situation  awareness 
teaching  has  been  the  IP  led  discussions  or  “walk-throughs.” 

This  method  requires  considerable  time  to  set  up  and  the  availability  of  an 
instructor  to  conduct  the  training.  With  advancements  in  technology,  SA  training 
should  be  able  to  immerse  the  PUI  in  a  digital  virtual  environment  that  simulates 
the  CAS  mission  environment.  The  virtual  training  environment  must  be  both 
expedient  for  the  instructor  to  set  up  and  easily  and  quickly  accessible  for  the 
student  to  utilize.  This  virtual  environment  can  then  provide  some  of  the  elements 
to  train  SA  that  a  PUI  would  only  receive  from  flying  actual  CAS  training  flights  in 
the  past. 

An  additional  consideration  needs  to  be  proven:  it  must  be  determined  if 
the  tasks  required  to  measure  SA  are  the  types  of  tasks  that  can  be  learned 
effectively  in  a  virtual  environment.  Immersion  and  interactivity  are  important  for 
the  PTT  VE  (Stanney,  Morurant,  and  Kennedy  1998,  333),  but  the  measurement 
of  PUI  success  will  be  gauged  from  the  scoring  of  the  SA  questions  they  answer. 
Even  though  the  PTT  as  implemented  by  Attig  (Attig  2016)  is  not  a  high  fidelity 
VE,  we  believe  it  should  be  capable  of  improving  the  SA.  An  additional  argument 
that  supports  this  understanding  is  a  study  by  John  et  al.  (John  et  al.  2015,  156). 
This  study  found  that  users  of  a  surgical  application  presented  on  a  tablet 
computer  experienced  positive  effects  performing  cognitive  tasks,  even  though 
the  application  was  considered  a  low  fidelity  simulation.  Also,  the  research  by 
Endsley  (2000)  found  that  cognitive  processes  were  very  important  in  acquiring 
and  maintaining  SA. 
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A  PTT  could  therefore  provide  the  environment  that  builds  a  mental  model 
of  the  CAS  battlefield  for  a  new  pilot  while  enabling  the  training  of  procedures. 
Endsley  surmises  these  mental  models  can  help  form  long-term  memory 
structures  for  the  pilots,  which  can  be  recalled  at  a  future  time  and  help  with 
situation  recognition  and  decision-making  (Rouse  &  Morris  1985,  27).  These 
mental  models  could  allow  the  pilot  to  recognize  situations  and  then  use  that 
long-term  memory  to  make  appropriate  timely  decision  (Endsley  2001 , 7). 
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SITUATION  AWARENESS  TRAINING  AND  MEASUREMENT 


A.  INTRODUCTION 

The  previous  chapters  defined  situation  awareness,  explained  its 
importance,  and  surmised  that  a  tablet  based  part  task  trainer  would  be  a  good 
platform  to  host  CAS  SA  training.  Before  implementing  this  PTT,  we  must  identify 
what  is  needed  to  train  CAS  situation  awareness  to  new  pilots  and  the  best  way 
to  measure  how  effective  the  SA  training  was  after  it  is  complete.  Former  Chief 
Scientist  for  the  U.S.  Air  Force  and  President  of  SA  Technologies,  Dr.  Mica 
Endsley,  has  dedicated  several  decades  to  studying  situation  awareness 
including  training  and  evaluating  SA;  much  of  that  work  was  focused  on  military 
aviation  and  tactical  environments.  We  used  many  of  her  findings  to  inform  the 
design  of  the  CAS  PTT  that  would  train  and  evaluate  situation  awareness. 

B.  WHAT  IS  NEEDED  TO  TRAIN  SA? 

One  of  the  first  tasks  in  determining  what  is  needed  to  train  SA  is  deciding 
what  types  of  requirements  are  related  to  situation  awareness.  Situation 
awareness  requires  more  dynamic  knowledge  than  traditional  static  knowledge 
job  requirements.  Situation  awareness  has  goal  oriented  objectives  that  require  a 
level  of  understanding  followed  by  a  decision  making  process  to  achieve  that 
goal.  Whereas,  static  jobs  require  more  following  rules  and  procedures  to 
complete  tasks.  (Endsley  2001 , 8).  These  static  tasks  frequently  instruct  the  user 
to  do  one  thing  then  another  in  a  certain  order  or  repeat  rote  knowledge. 
Dynamic  tasks  give  a  goal  state  then  require  the  user  to  interpret  information  and 
make  decisions  in  order  to  achieve  that  goal  rather  than  just  follow  a  set  of 
instructions.  Endsley  uses  a  goal  driven  task  analysis  format,  which  is  depicted  in 
Table  1. 
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Table  1 .  Format  of  Goal- Directed  Task  Analysis. 
Source:  Endsley  (2001). 


Goal 


Subgoal 

Decision 

Projection  (SA  Level  3) 
Comprehension  (SA  Level  2) 
Data  (SA  Level  1 ) 


Per  Endsley’s  definition,  SA  level  1  is  how  the  subject  perceives  the 
elements  of  the  environment.  SA  level  2  is  what  the  subjects  comprehension  of 
the  environment  is,  and  SA  level  3  is  how  the  subject  can  interpret  what  he  (she) 
comprehends  and  projects  what  the  future  outcome  will  be  (Endsley  2001,  4).  A 
modified  version  of  this  concept  will  be  used  to  determine  what  our  system  needs 
to  train  with  regard  to  SA. 

The  overall  goal  of  this  trainer  is  to  teach  the  situation  awareness  items 
that  are  important  for  safe  and  effective  CAS.  Subgoals  needed  to  support  this 
would  be  the  items  such  as  getting  ordnance  effects  on  the  target  and  not  on 
friendly  forces  (friendlies).  Knowing  where  the  enemy  and  friendlies  are  relative 
to  the  attacking  aircraft  is  needed  to  make  a  decision  and  accomplish  this 
subgoal.  Since  the  training  audience  for  this  trainer  consists  of  new  pilots,  the 
emphasis  will  be  on  training  the  SA  level  1  and  2  only.  New  pilots  need  to  have 
CAS  environmental  elements  presented  to  them  and  then  understand  those 
elements  in  the  environmental  context  rather  than  repeat  static  knowledge  items. 
Situation  awareness  accumulates  over  the  course  of  a  mission  and  it  is  therefore 
important  to  determine  what  elements  are  relevant  to  the  pilot.  The  pilot  has  to 
prioritize  the  information  and  then  use  it  to  make  decisions  based  on  projections 
of  the  near  future  (Endsley  1988,  791).  Even  though  the  focus  of  this  trainer  is 
the  lower  levels  of  SA,  the  situations  presented  in  it  and  experiences  of  the  pilots 
will  provide  a  foundation  for  more  advanced  SA  functions.  In  the  PTT,  the  pilot 
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will  be  provided  with  a  variety  of  environmental  elements  that  are  characteristic  of 
the  CAS  battlespace. 

There  is  no  evidence  of  any  direct  guidance  on  what  environmental 
elements  the  United  States  military  includes  in  situation  awareness  for  close  air 
support.  However,  connecting  the  link  between  situation  awareness  and  risk 
assessment  there  are  parallels  that  give  indirect  guidance  on  important  elements. 
Former  Ohio  State  University  Department  of  Aviation  Professor  Richard  Jensen 
linked  what  he  termed  aeronautical  decision  making  or  aviation  judgment  to  SA 
(Jensen  1997,  262).  He  stated  that  better  aeronautical  decision-making  leads  to 
better  SA  and  listed  five  main  factors  of  expert  aviation  judgment.  Three  of  those 
have  a  direct  impact  on  situation  awareness:  attention  control,  dynamic  problem 
solving,  and  risk  management.  Attention  control  and  dynamic  problem  solving 
are  straightforward.  Keep  focused  and  solve  problems  in  the  ever  changing  flight 
environment,  but  a  key  part  of  the  problem  solving  includes  being  able  to  use 
time  critical  pattern  matching  to  make  crucial  decisions.  This  is  very  similar  to  the 
pattern  matching  in  situation  awareness  (Endsley  2000,  6). 

Yet  according  to  Jensen’s  definition,  risk  management  is  the  most  similar 
to  situation  awareness  (Jensen  1997,  264).  He  explains,  “Risk  management 
means  being  aware  of  the  risks  in  all  flying  situations  and  being  able  to  assess 
them”  and  “ranking  them  least  risky  to  most  risky.”  He  also  adds,  “Risk 
assessment  requires  knowledge  and  SA.”  Being  aware  of  the  situation  or 
environment,  being  to  comprehend  or  assess  the  elements,  and  prioritize  them  in 
order  to  make  future  decisions  is  very  close  parallel  to  Endsley’s  view  on  SA 
(Endsley  1988,  791). 

Drawing  this  parallel  between  SA  and  risk  assessment,  we  find  a  list  that 

gives  us  guidance  to  what  elements  to  include  in  the  trainer.  The  Joint  CAS 

publication  presents  a  tactical  risk  assessment  in  the  CAS  execution  chapter 

(CJCS  2003,  V-18).  It  states  as  the  battlefield  environment  changes  CAS 

terminal  controllers  and  aviators  must  make  continuous  risk  assessments.  In 

those  assessments,  they  should  consider  time,  information  flow  and 
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communications,  tracking  of  battlefield  entities  both  friend  and  foe,  threat 
information,  and  target  information  including  marking.  These  items  will  form  the 
basis  of  our  elements  for  inclusion  in  the  trainer  environment.  The  elements 
needed  to  create  the  CAS  trainer  can  be  divided  into  three  categories:  elements 
and  events  related  directly  to  CAS  execution,  elements  pertinent  to  the  CAS 
environment  inside  the  aircraft,  and  environmental  elements  outside  the  aircraft. 

The  initial  environment  elements  included  in  the  trainer  are  those  relating 
directly  to  the  CAS  execution.  These  physical  elements  consist  of  the  target,  the 
anti-aircraft  threat,  the  friendly  ground  unit,  supporting  arms  and  the  terminal 
controller.  The  informational  elements  are  the  route  of  flight,  the  friendly  /  enemy 
situation  update,  the  CAS  attack  brief,  target  mark,  and  clearance  to  fire.  The 
CAS  terminal  controller,  the  person  on  the  ground  directing  the  CAS  aircraft 
where  to  go  and  when  and  where  to  shoot,  communicates  the  informational 
elements  which  include  details  about  the  physical  elements  to  the  pilot.  All  of  the 
elements  other  than  marking  the  target  for  identification  are  generally  conveyed 
to  the  pilot  via  radio  communication.  The  target  mark  can  be  an  audio  or  visual 
cue.  For  this  training  scenario,  we  will  use  the  visual  cue,  as  that  is  the  most 
common  for  CAS  missions. 

The  next  set  of  elements  will  be  the  tools  that  pilot  uses  in  the  cockpit  of 
the  aircraft  to  conduct  CAS.  The  items  a  pilot  utilizes  in  the  aircraft  when 
conducting  a  CAS  mission,  minus  weapon  systems  and  electro-optical  sensors, 
were  selected  for  representation  in  the  trainer.  These  items  are  a  moving  map 
display,  a  compass  heading  indicator,  a  timer,  and  radio  communications.  Aircraft 
frequently  have  electro-optical  sensors  that  can  enhance  situation  awareness, 
but  these  systems  were  omitted  because  they  are  not  needed  to  teach  basic 
CAS  SA  fundamentals.  Weapons  systems  were  omitted  since  ordnance  delivery 
does  not  directly  affect  SA. 

Finally,  the  trainer  will  create  ambient  environmental  elements.  These 

elements  were  limited  the  few  items  required  to  create  a  virtual  environment.  The 

terrain  is  simulated  to  create  a  realistic  visual  flight  environment.  Rotor  noise  is 
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included  to  further  increase  the  realism  as  it  represents  an  ever-present 
background  noise  distraction  that  pilots  would  expect  to  hear  in  an  operational 
environment.  The  firing  of  artillery  and  the  firing  aircraft  ordnance  are  also 
included  to  confirm  the  points  in  time  when  these  events  take  place. 

After  determining  what  elements  were  to  be  included  in  the  trainer,  the 
next  task  focused  on  the  form  in  which  they  would  be  presented  to  the  PUI. 
Based  on  the  analysis  of  CAS  missions  and  example  mission  flows  from  the  Joint 
CAS  publication  and  the  Marine  Aviation  Weapons  and  Tactics  Squadron  1 
(MAWTS-1),  it  was  determined  that  a  large  amount  of  the  SA  building  information 
comes  from  radio  communications  that  take  place  during  a  normal  CAS  mission 
flow  (CJCS  2009,  V- 22)  (United  States  Marine  Corps  2013).  The  PTT  uses  the 
normal  flow  of  CAS  radio  communications  between  the  terminal  controller  and 
the  pilot  to  provide  the  majority  of  essential  spatial  and  time  information  for  the 
PUI.  Information  such  as  the  location  of  battlefield  entities,  direction  of  fire,  and 
time  for  events  were  passed  to  the  pilot  in  this  manner.  The  pilot  needs  to  retain 
this  information  with  the  same  time  constraints  just  as  if  he  or  she  was  in  the 
aircraft  conducting  a  CAS  mission.  The  visual  elements  are  used  primarily  for 
orientation  such  as  knowing  if  the  aircraft  is  on  the  correct  heading  based  looking 
at  the  simulated  terrain,  the  map,  and  the  heading  indicator.  Also,  the  target 
marks  used  by  the  PUI  help  locate  the  target  and  confirm  it  is  the  correct  target 
with  a  cross  check  of  the  map. 

Overall,  all  of  these  environment  information  elements  give  the  PUI 
enough  data  for  the  PUI  to  understand  what  is  happening,  when  it  is  happening, 
and  where  it  is  happening,  so  they  can  execute  the  CAS  procedures  correctly. 
The  next  step  is  to  check  if  the  pilot  comprehends  these  details  and  truly  knows 
what  is  going  on  around  them  and  what  will  happen  in  the  near  future  (Endsley 
2001,  4).  To  accomplish  this  we  must  create  a  method  for  the  pilot  to  express 
their  perception  and  comprehension  of  the  battlefield  and  then  develop  a  means 
to  measure  how  accurate  the  pilot’s  views  are. 
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C.  HOW  TO  ASSESS  AND  MEASURE  SA 

At  some  point  after  receiving  the  environmental  information  to  build  SA,  it 
is  necessary  to  check  how  good  the  PUI’s  SA  is.  To  evaluate  SA  effectively 
several  things  must  be  determined.  The  first  is  to  determine  the  set  of  questions 
that  the  subjects  should  be  asked  to  properly  evaluate  their  SA.  Secondly,  the 
format  to  ask  the  questions  and  finally,  the  timing  to  ask  those  questions  need  to 
be  determined. 

The  points  in  time  when  questions  should  be  asked  are  based  on  the  goal 
based  task  analysis  (Endsley  2000a,  134).  According  to  Endsley,  the  questions 
need  to  test  the  spatial  and  time  awareness  and  the  analytical  capability  of  the 
subject.  Those  questions  should  query  aspects  of  the  pilot’s  perception  or 
comprehension  of  the  environment,  and  the  information  the  pilot  submits  as  they 
pertain  directly  to  the  goal  or  subgoal  of  the  task.  Questions  that  will  be  asked 
need  to  ensure  the  pilot  can  accurately  place  all  friendly  and  enemy  entities  at 
their  correct  location  on  a  map  to  demonstrate  spatial  comprehension.  The  pilot 
will  be  asked  about  the  timing  of  artillery  firing,  aviation  attacks,  and  the  order  of 
events  on  the  battlefield.  The  questions  are  structured  to  reveal  what  the  PUI 
comprehends  about  the  entire  battle  space  and  to  ensure  that  he  (she)  does  not 
have  tunnel  vision  on  only  one  aspect,  which  is  a  frequent  pitfall  of  new  pilots. 
Currently,  IPs  commonly  ask  these  questions  during  “chalk  talks”  and  “walk 
throughs,”  so  these  questions  are  of  the  type  that  SMEs  would  use. 

The  way  in  which  questions  should  be  asked  to  best  evaluate  pilot’s  SA  is 
the  next  area  that  must  be  addressed.  Endsley  lists  several  types  of 
measurements  for  SA  (Endsley  1988,  792).  There  are  measures  that  evaluate 
the  subject’s  perceptions  directly  such  as  questionnaires  or  self-assessments 
and  there  are  indirect  measurements  such  as  behavioral  and  performance 
assessments.  The  indirect  measure  evaluates  how  well  the  subject  did  on  a 
certain  task  and  then  infers  that  this  success  was  due  to  the  level  of  SA. 
However,  overall  success  does  not  necessarily  mean  it  was  due  to  SA,  and  it 
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could  be  attributed  to  variety  of  other  factors.  Due  to  this  understanding,  we 
opted  for  using  a  direct  measurement  (Endsley  2001 , 4). 

Now  that  direct  measurement  has  been  decided  on,  the  next  step  is  to 
decide  how  to  do  that  direct  measurement.  There  are  subjective  and  objective 
ways  of  measuring  SA  (Endsley  1988,  792).  One  of  the  easier  ways  to  measure 
SA  directly  proposed  by  Endsley  is  to  ask  the  subject  how  they  feel  about  their 
SA  or  have  an  observer  rate  the  subject’s  SA.  This  creates  a  large  margin  of 
error.  Subjects  may  think  they  know  more  than  they  do  and  third  person  observer 
can  only  use  the  outcome  of  a  situation  to  determine  SA.  The  subject  may  have 
no  idea  what  is  going  on  around  them,  but  succeed  on  a  lucky  guess  and 
deceive  the  observer.  The  best  technique,  according  to  Strater  et  al.  (2001)  and 
Endsley  (1988),  is  to  use  objective  questioning  to  evaluate  SA. 

The  question  format  is  the  next  item  that  must  be  decided.  Two  formats 
are  proposed  for  use  in  this  trainer:  one  which  users  must  select  the  correct 
answer  from  a  list  of  possible  answers,  commonly  referred  to  as  multiple  choice, 
and  questions  where  the  user  inputs  a  unique  answer  like  fill  in  the  blank  or  short 
answer.  The  selection  type  question  is  better  for  measuring  SA.  These  types  of 
questions  are  easily  measured  because  the  questions  have  a  single  correct 
answer  and  can  be  statistically  analyzed  easily.  Also,  multiple  choice  questions 
should  be  quicker  for  the  user  to  answer  than  the  one  requiring  explanation  and 
longer  input.  Multiple  choice  questions  can  be  kept  objective  as  well. 

Deciding  on  the  points  in  time  when  questions  are  to  be  asked  is  one  of 

the  last  items  to  address.  The  Situation  Awareness  Global  Assessment 

Technique  (SAGAT)  developed  by  Mica  Endsley  provides  excellent  guidance  on 

how  and  when  to  ask  SA  evaluating  questions  during  training  (Endsley  1988, 

789).  Many  evaluations  are  set  up  to  ask  questions  at  the  completion  of  training, 

but  it  is  often  hard  to  recall  specific  details  well  after  their  occurrence  (Endsley 

1988,  793).  For  dynamic  situation  awareness  queries,  we  want  to  capture  the 

subject’s  comprehension  near  real  time.  Endsley’s  answer  is  to  “freeze”  the 

training  to  ask  the  question.  For  this,  the  PTT  will  pause  and  the  question  will 
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appear  on  the  PTT  screen.  The  PUI  answers  the  question  with  simulation  time 
stopped.  Stopping  the  simulation  time  prevents  the  question  from  interfering  with 
the  training  scenario  and  the  pilot’s  comprehension  at  exactly  that  time  is 
recorded. 

Finally,  the  results  of  the  PUI’s  answers  are  collected  and  recorded  so  an 
IP  can  review  them  and  evaluate  the  PUI’s  performance.  This  record  needs  to 
contain  sufficient  information  and  be  easily  readable  by  the  instructor  if  it  is  to  be 
effective.  The  output  of  the  PUI’s  performance  contains  the  following  information 
for  each  question:  the  question  number,  the  PUI’s  answer,  the  correct  answer, 
and  how  long  it  took  the  PUI  to  answer  the  question.  The  output  also  presents  a 
composite  score  based  on  the  total  number  of  questions  correct.  This  information 
provides  the  instructor  with  enough  information  to  see  where  the  PUI  is 
succeeding  or  struggling  with  comprehension  of  the  CAS  environment.  The 
record  should  also  print  out  in  a  format  that  is  easily  readable  by  the  instructor. 

D.  CHAPTER  SUMMARY 

In  this  chapter,  we  determined  how  to  use  the  PTT  to  present  elements  of 
the  CAS  environment  and  develop  situation  awareness  in  CAS  for  the  new  pilot. 
The  text  also  discussed  the  best  ways  in  which  pilot’s  SA  could  be  evaluated, 
format  in  which  that  is  done,  and  the  timing  of  those  queries. 
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IV.  TASK  ANALYSIS 


A.  INTRODUCTION 

The  CAS  operator  is  required  to  assimilate  and  sort  a  large  amount  of 
information,  and  it  makes  it  difficult  for  the  operator  to  effectively  organize  that 
data  to  gain  high  level  of  situation  awareness.  To  make  the  data  manageable, 
having  a  systematic  and  integrated  process  is  very  much  needed.  Task  analysis 
helps  understand  many  elements  specific  for  the  task  of  interest,  including  the 
environment  and  conditions  in  which  the  task  is  executed,  and  the  goals  that 
need  to  be  achieved  by  the  user.  Task  analysis  breaks  down  the  steps  of  the 
task,  and  how  an  individual  executes  each  step  specifying  the  conditions  and 
level  of  mastery  needed  to  achieve  task  goals  (the  same  approach  applies  if 
more  than  one  user  is  engaged  on  given  task).  Once  each  step  is  identified  and  a 
measurable  performance  specified,  the  supporting  system  architecture  and 
necessary  teaching  methods  can  be  developed. 

B.  EXECUTION 

Two  tasks  were  analyzed  in  support  of  this  research  effort.  Helicopter 
pilots  conduct  each  of  the  tasks,  one  pilot  is  an  instructor  (IP)  and  the  other  is  a 
trainee  (PUI).  The  PUI  is  a  novice  operator  while  the  IP  is  an  expert  operator  in 
the  Close  Air  Support  domain.  The  PUI  is  qualified  to  fly  and  operate  the  aircraft, 
but  has  had  only  limited  exposure  to  tactical  employment  like  CAS.  The  IP  has 
reached  the  expert  level  in  CAS  operations  through  several  years  of  dedicated 
training  and  dozens  of  CAS  training  and/or  real  world  CAS  missions.  Also,  the  IP 
has  completed  an  additional  training  syllabus  teaching  him  (her)  how  to  instruct 
CAS  training. 

Many  entities  are  involved  in  actual  CAS  operations.  They  all  have 
discreet  tasks  they  must  accomplish  to  execute  the  CAS  mission.  For  this  part 
task  trainer,  we  will  focus  on  the  tasks  required  by  the  PUI  and  IP:  the  creation 
and  execution  of  the  scenario  to  train  situation  awareness.  The  IP's  tasks  are 
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conducted  in  the  same  manner  as  actual  CAS  pre-mission  planning  would  be 
done.  This  planning  usually  happens  inside  a  unit's  planning  rooms  while  using 
mission  planning  tools  like  maps  and  diagrams  of  the  battle  area.  The  IP's 
responsibility  is  to  create  the  CAS  environment  for  the  PUI.  In  a  CAS  mission, 
training  or  real  world,  the  PUI  would  be  in  the  cockpit  of  the  aircraft  managing  the 
information  required  to  gain  SA  by  observing  the  battlespace  environment, 
listening  to  radio  communications,  and  operating  the  aircraft  systems.  This  would 
be  a  very  dynamic  environment  even  for  the  simple  introductory  missions.  Our 
intent  is  to  create  a  CAS  environment  with  few  distractors  while  maintaining  the 
time  constraints  and  environmental  information  required  for  CAS.  This  is  so  the 
PUI  can  focus  on  the  essential  tasks  required  to  gain  SA  using  the  simulated 
elements  of  the  PTT. 

C.  PREVIOUS  CAS  TASK  ANALYSIS  WORKS 

Attig  (2106)  completed  a  formal  task  analysis  focused  on  the  pilot's  tasks 
required  to  conduct  CAS  in  his  Master's  thesis.  His  task  analysis  used  the  AH- 
1W  T&R  manual  and  was  presented  in  the  behavioral  task  analysis  form  with 
individual  observable  actions  (Clark  and  Estes  1996,  403).  Each  task's  goals 
began  with  a  certain  initial  state,  which  precipitated  a  specific  action  by  the  pilot. 
This  produces  a  particular  results  and  then  on  to  the  next  sequential  task.  For 
instance,  prior  to  arriving  at  the  objective  area  (basically  the  battle  area  where 
the  kinetic  portions  of  CAS  take  place)  the  pilot  needs  to  tune  to  a  certain  radio 
frequency,  contact  the  terminal  controller,  and  expect  to  receive  instruction  of 
where  to  go  and  await  further  CAS  instructions  (Attig  201 6). 

Conversely,  the  task  analysis  for  situation  awareness  needs  to  be  a 
cognitive  task  analysis  because  SA  is  mental  process  requiring  cognitive  abilities. 
Clark  and  Estes  (1996)  stated  that  cognitive  abilities  include  items  such  as 
pattern  recognition,  perception,  attention,  and  decision  making,  all  of  which  have 
been  directly  associated  with  situation  awareness  (403).  As  stated  earlier, 
Endsley  (2001)  also  believes  SA  task  analysis  is  best  served  using  a  "goal- 
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directed"  cognitive  task  analysis  to  capture  the  dynamic  nature  of  SA.  (8)  This 
differs  from  the  behavioral  task  analysis  in  that  each  goal  oriented  task  is 
supported  by  a  decision  informed  by  the  operator's  three  levels  of  SA, 
perception,  comprehension,  and  projection  instead  of  following  a  sequential 
procedure. 

D.  COGNITIVE  TASK  ANALYSIS  FOR  CAS  SA 

The  task  analysis  for  this  study  will  be  divided  into  two  separate  cognitive 
task  analyses:  one  task  analysis  for  the  IP’s  task  of  building  of  the  CAS  scenario, 
and  one  for  the  PUI’s  task  of  conducting  the  CAS  scenario.  Building  the  CAS 
scenario  for  the  trainer  requires  a  cognitive  task  analysis  because  the  IP  has  to 
construct  the  scenario  using  complicated  problem-solving  skills  rather  than 
simply  following  step-by-step  instructions.  According  to  the  Combat  Aircraft 
Fundamentals  UH-1  guidelines,  the  IP  builds  the  CAS  scenario  very  similarly  to 
the  way  a  pilot  plans  a  CAS  mission.  (Department  of  the  Navy  2013).  The  IP  is 
an  expert  in  CAS  and  knows  how  to  build  the  scenario;  the  IPs  extensive 
experience  in  planning  and  executing  CAS  missions  is  extremely  important  in  this 
process.  The  IP  begins  by  developing  the  big  picture  of  the  battlespace 
environment,  and  then  continues  with  the  rest  of  the  specific  steps  aimed  at 
building  the  scenario. 

1 .  CAS  Scenario  Builder  Cognitive  Task  Analysis 

The  scenario  builder's  top  level  goal  is  to  create  the  CAS  scenario.  The  IP 
starts,  as  stated  earlier,  by  developing  the  big  picture  of  the  battlefield  first.  In 
military  verbiage  the  big  picture  is  often  termed  "big  blue  arrows,"  signifying  large 
blue  arrows  depicted  on  military  maps  that  represent  friendly  troop  movements 
across  the  battlefield.  This  "big  blue  arrow"  picture  is  the  overall  scheme  of 
maneuver  of  both  the  friendly  and  enemy  units.  For  example,  the  situation  may 
be  the  friendly  side  attacking  to  seize  a  key  road  intersection  with  the  enemy 
defending  it  or  the  enemy  trying  to  capture  an  important  industrial  city  and  the 
friendlies  holding  the  high  ground  outside  the  city.  The  scenario  starts  at  this 
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level:  the  friendly  ground  unit's  scheme  of  maneuver  and  fire  support  plan  as  well 
as  the  enemy  disposition  determines  how  the  CAS  mission  will  be  conducted 
(Department  of  the  Navy  2013).  These  will  be  the  major  subtasks  in  the  task 
analysis.  Once  these  items  are  determined,  further  subtasks  must  be  completed 
to  create  the  scenario.  The  CAS  joint  publication  describes  battle  tracking  as 
constructing  the  battle  environment  in  an  accurate  manner  (CJCS  2009,  V-2). 
The  elements  of  battle  tracking  comprise  the  essential  elements  the  IP  needs  to 
include  in  the  scenario  build.  Those  include 

1 .  Fire  Support  Control  Measures  (FSCMs)  and  Airspace 

Coordination  Measures  (ACMs) 

2.  Friendly  Unit  Location 

3.  Artillery  Location,  Gun  Target  Lines  (GTLs)  and  targets 

4.  Target  Location 

5.  Orders  like  CAS  attack  briefs 

6.  Communications 

Integration  of  these  components  allows  PTT  operator  to  create  a  CAS 
scenario  for  the  PUI.  We  have  limited  the  tasks  and  conditions  of  CAS  operations 
supported  in  the  PTT:  this  includes  only  day,  Type  1  CAS  requirements  for  non¬ 
precision  guided  ordnance.  Additionally  it  does  not  cover  any  aircraft,  ordnance, 
or  targeting  specifics.  Table  2  provides  details  of  the  complete  scenario  building 
task  analysis. 
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Table  2.  IP  Cognitive  Task  Analysis  for  CAS  Scenario  Creation. 


1 .  Create  CAS  scenario. 

1 .1 .  Determine  friendly  and  enemy  scheme  of  maneuver  in  terms 
of  each  side's  objectives  during  the  scenario. 

1.1.1.  Determine  location  of  objective  area  based  on  terrain. 

1 .1 .2.  Determine  Enemy  disposition. 

1 .1 .3.  Determine  Friendly  disposition. 

1 .2.  Build  objective  area 

1.2.1.  Determine  and  place  target  location. 

1 .2.2.  Determine  and  place  threat  location. 

1 .2.3.  Determine  friendly  location. 

1 .2.4.  Determine  and  place  Artillery  Gun  Positions 
(GPs). 

1 .2.5.  Determine  and  place  Artillery  at  desired  GP. 

1 .3.  Create  Fire  Support  Plan  in  support  of  Friendly  scheme  of 
maneuver. 

1 .3.1 .  Create  FSCMs  and  ACAs  with  regard  to  Friendly  and 
Enemy  positions. 

1 .3.1 .1 .  Place  Battle  Positions  (BPs). 

1.3. 1.2.  Place  Holding  Areas  (HAs). 

1 .3.2.  Create  aircraft  route  to  CAS  objective  area. 

1.3. 2.1.  Select  route  checkpoints. 

1 .3.3.  Create  9  Line  CAS  attack  brief. 

1.3. 3.1.  Determine  and  select  BP. 

1.3. 3. 2.  Determine  and  select  HA. 

1 .3.3.3.  Determine  and  select  aircraft  Egress. 
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2. 


CAS  Scenario  Execution  for  Situation  Awareness  Cognitive 
Task  Analysis 


The  second  part  of  our  task  analysis  captures  the  PUI  task  of  using  the 
PTT  to  execute  CAS  mission  (scenario).  Most  of  the  pilot's  procedural  tasks 
described  in  Attig's  (2016)  task  analysis  are  conducted  automatically  by  the  PTT. 
Still,  the  PUI  is  expected  to  understand  these  steps  and  to  know  when  and  where 
they  should  occur.  The  PUI  is  also  responsible  for  all  of  the  information  provided 
in  these  steps;  PUI  needs  to  retain  it  for  use  during  the  scenario  as  that  same 
information  drives  the  PUI's  decision  making  process.  Much  of  this  information 
will  be  passed  via  simulated  radio  communications  during  the  scenario. 

The  focus  of  this  task  analysis  is  on  the  cognitive  tasks  required  by  the 
pilot  to  attain  and  maintain  situation  awareness.  Each  of  the  steps  involves  the 
PUI  understanding  of  the  elements  in  the  aforementioned  CAS  joint  publication's 
tactical  risk  assessment  (CJCS  2009,  V-18).  This  task  analysis  limits  its  elements 
to  those  required  for  initial  CAS  training  and  elements  that  are  present  in  the 
CAS  PTT:  CAS  check-in,  CAS  situation  update,  CAS  9  Line  attack  brief  with 
supporting  arms  and  threat  remarks,  and  time-on-target  (TOT)  for  Type  1  attack. 
As  commented  earlier,  the  PTT  and  corresponding  task  analysis  do  not  cover  the 
full  realm  of  CAS  possibilities.  Also,  more  than  one  task  or  subtask  may  be  active 
at  the  same  time  and  different  tasks  may  share  subtasks.  Table  3  presents  all 
details  of  the  CAS  Execution  for  SA  cognitive  task  analysis. 
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Table  3.  CAS  Execution  for  SA  Cognitive  Task  Analysis. 


1 .  Perceive  and  comprehend  the  CAS  battlespace 

1 .1 .  Determine  how  friendly  elements  affect  the  CAS  attack. 

1.1.1.  Determine  how  to  support  friendly  ground  units  and 

avoid  fratricide. 

1.1. 1.1. 

Know  the  distance  of  the  friendly  maneuver 

element  from  the  target. 

1.1. 1.2 

Compute  what  final  attack  headings  (FAHs) 

are  usable  from  the  selected  BP. 

1.1. 1.3. 

Know  effects  distance  of  CAS  fires. 

1.1. 1.4. 

Know  target  location. 

1.1. 1.5. 

Know  aircraft  location. 

1.1.2.  Determine 

how  supporting  arms  affect  the  CAS  attack? 

1. 1.2.1 

Know  artillery  location. 

1.1. 2. 2. 

Know  artillery  GTL  and  what  if  any  ACMs  or 

FAHs  it  crosses. 

1.1. 2. 3. 

Know  when  and  where  artillery  is  firing 

suppression. 

1.1. 2. 4. 

Know  when  artillery  is  marking  the  target. 

1.1. 2. 5. 

Know  aircraft  location. 

1 .2.  Determine  how  enemy  elements  affect  the  CAS  attack. 

1.2.1.  Determine  how  the  target  impacts  the  CAS  attack. 

1. 2.1.1. 

Know  target  location. 

1.2. 1.2. 

Know  target  type. 

1 .2.2.  Determine  how  and  where  the  air  threat  impacts  the  CAS 

attack. 

1.2. 2.1. 

Know  threat  location. 

1.2. 2. 2. 

Know  threat  range. 

1.2. 2. 3. 

Know  aircraft  location. 
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1 .2.3.  Resolve  how  threat  will  be  mitigated  during  attack. 

1 .3.  Determine  CAS  attack  from  selected  ACMs  is  tactically  sound. 

1 .3.1 .  Know  attack  heading  from  BP  to  target. 

1 .3.2.  Ensure  that  FAHs  point  toward  the  target  and  do  not 
encroach  on  GTL  or  head  toward  friendlies. 

1 .3.3.  Ensure  that  target  mark  is  visible  on  ingress  from  BP  to 
target. 

1 .3.4.  Ensure  that  route  of  flight  avoids  threat  and  GTL. 

1 .4.  Determine  how  time  affects  the  CAS  attack. 

1 .4.1 .  Know  when  you  should  receive  routing  and  safety  of  flight, 
situation  update,  CAS  attack  brief,  remarks  and  restrictions, 
TOT,  and  clearance. 

1 .4.2.  Determine  when  suppression  will  be  impacting  target. 

1 .4.2.1.  Know  TOT. 

1 .4.3.  Determine  when  target  mark  will  be  visible. 

1 .4.3.1.  Know  TOT. 

1 .4.4.  Determine  time  to  move  from  HA  to  BP  to  meet  TOT. 

1.4. 4.1  Know  current  time. 

1.4. 4. 2.  Know  TOT. 


E.  CHAPTER  SUMMARY 

This  chapter  presented  the  results  of  two  cognitive  task  analyses  in 
support  of  PTT  for  CAS  situation  awareness;  the  analysis  augmented  previously 
executed  behavior  task  analysis  for  CAS  execution  (Attig  2016).  The  first 
cognitive  task  analysis  was  focused  on  IP’s  creation  of  the  CAS  SA  scenario,  and 
the  second  cognitive  task  analysis  was  focused  on  Pill’s  execution  of  the  CAS 
SA  scenario. 
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V.  SYSTEM  DESCRIPTION 


A.  FRAMEWORK 

The  HMLA  CAS  PTT  for  SA  was  created  using  a  part  task  trainer  proof  of 
concept  model  designed  by  Attig  (2016).  Attig  used  the  Unity  game  engine  to 
develop  all  necessary  elements  of  his  proof  of  concept  PTT.  We  used  the  same 
framework  architecture  to  support  IP’s  CAS  scenario  building  and  PUI’s  CAS 
scenario  execution.  The  IP  builds  the  scenario  using  the  capabilities  of  the  IP 
interface;  that  scenario  is  then  exported  as  a  JavaScript  Object  Notation  (JSON) 
file.  The  JSON  file  is  imported  and  run  using  the  PUI  interface.  Figure  1  depicts 
the  elements  of  system  architecture  including  data  flows  and  user  interaction. 


Figure  1.  HMLA  CAS  PTT  Architecture.  Adapted  from  Attig  (2016). 
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Some  elements  from  Attig's  (2016)  master’s  thesis  project  were  used  in 
our  trainer.  They  included  the  terrain,  3D  model  of  the  Cobra  helicopter,  the 
question  and  answer  interface,  and  the  system  to  convert  latitude  and  longitude 
coordinates  to  Unity  coordinates.  The  rest  of  the  system  needed  to  support  the 
specifics  of  SA  elements  was  designed  and  implemented  by  the  author  of  this 
thesis.  Those  efforts  focused  on  adding  the  additional  entities,  events,  and 
information  required  to  create  the  CAS  battlefield  and  to  create  a  situation 
awareness  enriching  environment. 

B.  DEVELOPMENT  AND  PROGRAMMING  ENVIRONMENT 

The  Unity  game  engine  was  chosen  for  the  CAS  PTT  for  several  reasons. 
First,  Unity's  universal  capabilities  are  a  large  reason  it  was  selected.  Unity 
provides  the  flexibility  to  run  on  a  wide  range  of  platforms  including  personal 
computers,  tablet  computers,  and  smart  phones  on  a  multitude  of  different 
operating  systems.  The  Unity  software  can  run  locally  as  an  independent  (stand¬ 
alone)  application  or  be  web  based  and  run  from  the  cloud.  Also,  Unity  is 
compatible  with  a  large  array  of  audio  and  video  files,  and  Unity  has  a  large 
library  of  3D  models  and  software  development  kits  available  for  use. 

Applications  can  be  started  quickly  in  Unity  due  to  the  extensive  tutorials 
provided  by  Unity  and  the  intuitive  game  development  environment.  Projects  can 
be  developed  in  Unity  using  the  graphic  editor  interface,  via  scripting,  or  a 
combination  of  both.  See  Figure  2  to  view  the  Unity  graphic  editor.  Unity  comes 
with  the  MonoDevelop  integrated  development  environment  (IDE)  to  write  scripts 
and  compatible  with  many  other  IDEs.  For  the  CAS  PTT  we  used  a  combination 
of  the  graphical  editor  and  scripting  with  MonoDevelop.  The  graphical  editor  was 
used  for  building  the  CAS  PTT  graphic  user  interface  (GUI)  and  visual  3D 
models.  Scripting  in  MonoDevelop  created  all  the  CAS  PTT  behaviors  and 
calculations. 
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Figure  2.  Unity  Graphic  Editor. 
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C.  INTERFACE  FOR  SCENARIO  BUILDER  (IP  INTERFACE) 

To  begin  creating  the  complete  CAS  scenario  we  needed  to  add  critical 
physical  elements  and  their  associated  behaviors  to  the  scenario.  These 
elements  include  the  target,  the  friendlies,  the  artillery,  and  a  threat.  These 
entities  need  to  be  placed  in  the  virtual  environment  at  specific  locations  by  the 
IP.  To  do  this,  the  IP  needed  an  effective  way  to  insert  these  entities  at  exact 
geographic  coordinates  and  then  see  a  visualization  to  ensure  the  placement 
was  correct.  We  decided  to  use  a  map  system  to  allow  the  IP  to  view  the 
placement  of  the  battlefield  entities'  locations  -  the  interactive  style  used  for  this 
part  of  the  interface  was  "direct  manipulation."  The  IP  positions  the  entities  at 
locations  on  a  map  and  these  positions  translate  directly  to  corresponding 
locations  on  the  3D  terrain  that  the  PUI  views.  To  position  the  entities  precisely, 
we  decided  to  utilize  a  gridded  2D  map,  as  it  is  the  most  common  spatial 
reference  representation  used  by  pilots  in  CAS  missions.  Since  the  terrain  used 


33 


in  the  CAS  scene  represented  the  Marine  Corps  Air  Ground  Combat  Center 
(MCAGCC)  at  Twentynine  Palms,  California,  the  Twentynine  Palms  Military 
Installation  Map  (MIM)  was  used  to  depict  that  same  area  in  the  IP  interface. 
(National  Geospatial-Intelligence  Agency  (NGA)  2009).  The  2D  Twentynine 
Palms  MIM  is  a  much  lower  resolution  than  the  terrain  at  the  1:100,000  scale, 
meaning  1  cm  of  distance  on  the  map  represents  1  km  on  the  real  ground.  This  is 
sufficient  detail  to  depict  the  entities  in  order  to  reference  their  location  while  the 
3D  terrain  is  a  much  higher  resolution  to  accurately  depict  the  terrain  detail  for 
visualization  in  the  scenario  execution.  The  entities  were  then  represented  on  the 
map  with  military  operational  graphic  icons.  This  allowed  the  IP  to  know  the  exact 
real  world  coordinates  for  every  entity. 

The  technical  skills  of  IPs  may  differ  greatly.  Some  may  have  extensive 
computer  skills,  and  some  may  have  very  basic  or  limited  computer  skills.  To 
support  IPs  with  extensive  CAS  skills  but  who  may  have  basic  or  limited 
computer  skills,  it  was  decided  to  further  streamline  and  simplify  this  IP  interface, 
and  make  sure  that  all  IPs  can  create  CAS  scenarios  in  a  timely  manner.  The 
use  of  the  map  in  the  IP’s  interface  was  the  first  step  in  that  direction.  The  next 
step  was  to  create  the  CAS  situation  and  attack  brief  that  would  drive  the 
scenario.  To  do  this  we  made  use  of  the  traditional  9  Line  CAS  attack  brief 
complete  with  remarks  -  this  brief  provides  the  most  familiar  and  succinct  format 
to  generate  the  CAS  scenario.  It  contains  all  of  the  essential  elements  and  their 
specifics:  the  target,  friendlies,  artillery,  and  threat.  CAS  scenarios  can  be  very 
complicated  and  require  more  information,  but  the  9  Line  CAS  attack  brief  and 
map  depicting  the  essential  entities  locations  are  suitable  for  the  basic  CAS 
mission  used  to  train  a  new  pilot.  Using  the  capabilities  of  the  9  Line  and  the 
map,  an  IP  could  now  easily  create  the  entire  CAS  scenario  in  IP  interface. 

Originally,  we  designed  the  interface  so  the  IP  would  need  to  input  the 
coordinates  for  the  entities,  ACMs,  route  waypoints  and  the  CAS  9  Line 
information  by  tying  in  each  location.  After  completing  the  input,  the  IP  would 
load  them  to  the  map  and  see  the  location  of  each  entity.  Our  initial  tests  showed 
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that  this  process  was  time  consuming.  Also,  it  was  easy  to  type  an  incorrect 
number  and  not  realize  it  until  the  all  data  entry  is  complete  and  when  viewing 
the  map.  To  improve  this  capability,  it  was  decided  to  make  the  scenario 
generation  similar  to  real  CAS  planning  and  make  sure  that  this  process  is  as 
intuitive  as  possible. 

In  actual  CAS  planning,  once  the  ACM  and  entity  locations  are  known, 
they  are  plotted  on  the  map.  To  mimic  this  we  forgo  inputting  the  entities  and 
ACMs'  coordinates  manually,  and  instead  allowed  the  IP  direct  manipulation  of 
the  icons — IP  can  drag  and  drop  the  objects  representing  the  same  entities  to  the 
correct  locations  on  the  map.  The  IP  gets  immediate  visual  feedback  about  the 
location  and  Unity  scripts  then  calculate  the  coordinates  of  all  objects.  Similarly, 
the  route  waypoints  are  selected  instead  of  entering  their  coordinates  (numbers) 
manually.  Route  waypoints  are  usually  predetermined  checkpoints  stored  on  a 
list.  Most  likely  they  are  selected  during  a  planning  stage  that  helps  create  a 
route  to  get  the  aircraft  to  the  objective  area  quickly  while  avoiding  other  air  traffic 
and  threats.  The  interface  was  changed  to  accept  a  list  of  waypoints  from  the 
commonly  used  and  easily  compatible  comma-separated  value  (CSV)  file.  The 
CSV  file  contains  information  about  each  checkpoint  name,  its  latitude  and 
longitude  common  to  most  readily  available  waypoint  lists.  The  waypoints 
populate  a  drop  down  selection  box  in  the  interface  and  the  IP  selects  the 
waypoints  as  desired.  The  selected  waypoints,  entity  locations,  and  ACMs  then 
generate  the  latitude  and  longitude  coordinates  for  the  JSON  file  automatically. 
Each  waypoint  and  the  route  are  displayed  on  the  map  as  the  IP  selects  them. 
Figure  3  shows  the  elements  of  the  graphic  interface  that  enables  building  CAS 
scenario  objective  area. 
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Figure  3.  CAS  Objective  Area  Builder  Map  Interface. 


The  9  Line  builder  interface  displays  the  9  Line  CAS  attack  brief  to  the  IP. 
Originally,  the  9  Line  builder  required  the  IP  to  enter  a  large  amount  of 
information  via  drop  down  selections  and  type  text  inputs.  The  IP  needed  to 
reference  the  location  of  the  entities  and  ACMs  from  the  map  and  make  the 
entries  that  consumed  a  lot  of  time  and  increased  the  opportunity  for  error.  After 
review  and  the  implementation  of  the  drag  and  drop  entities  and  ACMs,  we 
decided  to  translate  the  location  coordinates  of  the  objects  on  the  map  into 
numerical  data  and  use  that  to  auto-populate  select  elements  of  9  Line  form. 
Upon  activating  the  9  Line  builder,  it  now  displayed  the  target,  threat,  friendly, 
and  artillery  information  generated  by  the  objective  area  builder.  Selecting  the  BP 
for  the  CAS  aircraft  to  attack  from  the  drop  down  list,  the  system  calculates  the 
remaining  information  for  the  attack  brief  including  the  heading  and  distance  from 
the  selected  BP  to  the  target  from  the  location  data.  The  IP  then  selects  the 
aircraft  egress  direction  determining  the  direction  of  the  aircraft  pull  off  from  the 
target,  and  selects  the  HA  to  use.  This  information  is  then  saved  and  exported 
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via  the  CSV  file  so  that  it  can  be  used  in  the  CAS  execution  portion  (PUI 
interface).  Figure  4  shows  the  design  of  the  CAS  9  Line  builder  interface.  After 
building  the  scenario,  the  CSV  is  loaded  in  the  CAS  execution  portion  via  PUI 
interface.  The  JSON  file  method  is  used  so  IPs  can  build  multiple  scenarios  and 
save  each  as  a  separate  file.  This  helps  create  a  library  of  scenarios  for  the  PUI 
to  train  on. 


Figure  4.  CAS  9  Line  Builder  Interface. 


D.  CAS  SCENARIO  EXECUTION  INTERFACE  (PUI  INTERFACE) 

While  the  CAS  execution  scenario  looks  similar  to  the  one  in  Attig's  (2016) 
work,  several  changes  were  made  to  this  interface  to  make  sure  that  scenario 
provides  a  better  opportunity  for  acquiring  situation  awareness.  The  first  and 
most  noticeable  change  to  the  scene  is  the  addition  of  the  friendly  and  enemy 
units.  Three-dimensional  models  from  the  Unity  asset  store  representing  the 
target,  threat,  artillery,  and  friendly  maneuver  element  are  inserted  on  the  terrain 

for  visual  cueing.  The  appropriate  model  spawns  at  the  location  corresponding  to 
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where  the  IP  placed  during  the  scenario  building.  The  3D  models  add  to  the 
realism  of  the  CAS  scenario.  The  PUI  can  now  see  visible  representations  of 
those  objects.  Figure  5  depicts  3D  models  of  all  possible  entities. 


Figure  5.  3D  Modes  in  CAS  Execution  Scenario. 


We  created  effects  to  simulate  the  entities  interacting  on  the  battlefield 
environment.  For  example,  the  helicopter  engagement  sequence  on  the  target, 
the  artillery  firing  threat  suppression,  and  the  target  mark  are  visually  simulated 
events.  Also,  we  also  simulate  the  helicopter  firing  the  rockets  and  the  artillery 
muzzles  flash  when  firing  to  increase  the  visual  realism  of  the  training 
environment  and  improve  overall  user  experience.  Flashes  and  dust  clouds 
simulate  the  artillery  suppression  impacts  on  the  enemy  air  threat,  and  a 
simulated  explosion  shows  the  destruction  of  the  target.  Figure  6  shows  some  of 
those  visual  effects. 
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Figure  6.  Scene  with  Added  Visual  Effects. 


To  further  enhance  the  visual  realism  of  the  training  environment,  we 
changed  the  helicopter  flight  path  from  a  constant  altitude  to  a  path  that  follows 
the  contours  of  the  terrain.  This  flight  regime  is  more  authentic  to  the  actual  flight 
path  of  a  helicopter  that  performs  CAS  in  this  threat  level  environment.  The  lower 
altitude  and  terrain  conformance  restricts  the  pilot's  line  of  sight  distance  and 
increases  the  pilot's  dependency  on  radio  communications  to  provide  SA  details. 
We  accomplished  this  by  raycasting  along  the  flight  path  to  derive  a  height  of  the 
helicopter  above  terrain  altitude.  Other  elements  that  were  added  to  enable  SA 
stimulus  included  the  capability  for  the  pilot  to  pivot  the  camera  view,  which 
simulated  head  turning  of  the  pilot,  and  allow  for  independent  view  of  the  virtual 
environment. 

Also,  we  added  several  SA  buildering  tools  that  the  pilot  would  normally 
have  in  the  real  cockpit.  A  digital  clock  was  added  to  keep  track  of  mission  timing 
and  ensure  meeting  the  TOT.  A  compass  with  a  magnetic  heading  indicator  was 
also  added  to  inform  the  PUI  what  direction  he  (she)  is  going.  This  allows  the 
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pilot  to  reference  the  aircraft  heading  as  it  approaches  the  target,  and  ensure  that 
the  aircraft  is  within  the  bounds  of  the  authorized  final  attack  headings  (FAH). 
These  directional  heading  restrictions  given  by  the  terminal  controller  are  to 
guarantee  the  aircraft  only  shoots  when  pointed  in  a  safe  direction  and  it  is  very 
important  for  the  pilot  to  know  that  he  or  she  is  within  the  limits.  Every  pilot 
always  has  a  map  in  the  cockpit,  so  the  same  map  that  is  used  by  the  IP  to  help 
create  CAS  scenario  is  included  in  this  interface  as  well.  The  PUI  is  able  to 
switch  the  map  on  and  off  as  needed.  The  map  can  be  used  to  correlate  the 
PUI's  position  above  the  virtual  terrain.  Additionally,  a  moving  icon  representing 
the  helicopter's  position  is  available  in  current  moving  map  displays  in  modern 
aircraft.  The  PUI's  map  is  also  designed  to  contain  the  same  ACMs  as  the  IP 
map;  the  only  difference  that  we  introduced  was  that  the  entities  are  not  initially 
placed  at  the  correct  locations.  During  the  course  of  CAS  scenario,  the  PUI  will 
receive  the  locations  of  these  entities  via  radio  communication  and  he  (she)  is 
requested  to  drag  the  entity  icons  to  the  correct  spot  on  the  map.  This  allows 
testing  PUIs  situation  awareness  in  context  of  the  given  scenario.  Figure  7  shows 
PUI  interface  with  the  cockpit  SA  builders. 

To  improve  the  SA  questions-and-answers  experience,  we  added 
questions  that  require  PUI  to  drag  and  drop  the  entities  to  their  corresponding 
locations  on  the  map.  PUIs  are  graded  on  their  ability  to  place  the  entity  icons  at 
the  correct  location  (the  information  about  those  locations  is  passed  in  the 
terminal  controller's  situation  update  and  attack  brief.)  Other  questions  are  used 
to  evaluate  the  PUI's  SA.  They  consist  of  multiple  choice  questions  placed  in  a 
communications  /  data  unit  (CDU),  depicted  in  the  lower  left  corner  on  Figure  7. 
During  the  simulation,  the  CDU  pop-ups  in  the  lower  left  corner  and  displays  SA 
questions  that  the  IP  created.  The  text  font  size  in  initial  design  of  CDU  was  too 
small  to  be  easily  read,  and  the  touch  screen  buttons  on  the  CDU  were  too  small 
for  larger  fingers  to  activate  quickly  on  the  first  tap.  We  decided  to  increase  the 
overall  size  of  the  CDU  interface  and  alleviate  those  problems.  Figure  8  shows 
changes  that  were  made  with  design  of  CDU. 
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Figure  7.  CAS  Execution  Scenario  Cockpit  SA  Builders. 
Adapted  from  Attig  (2016). 


Figure  8.  Increased  Size  of  Question  and  Answer  Interface. 

Adapted  from  Attig  (2016). 


The  PUI's  answers  to  the  questions  are  recorded  to  another  CSV  file  at 
the  end  of  the  training  session.  This  CSV  output  file  contains  the  ID  of  each 
question,  the  time  stamp,  the  PUI's  response  to  that  question,  the  correct 
response,  the  time  it  took  the  PUI  to  answer  each  question  and  the  PUI's  overall 
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score.  All  of  this  is  useful  when  evaluating  the  Pill's  performance  and 
determining  the  focus  areas  for  future  training. 

E.  AUDIO  PRODUCTION 

Much  of  the  SA  information  the  PUI  receives  is  audio.  This  SA  building 
audio  is  almost  exclusively  in  the  format  of  simulated  radio  communications  in 
the  PTT.  The  IP  records  a  set  of  radio  communications  during  the  scenario 
development.  For  the  demo  scenario,  each  radio  call  was  recorded  individually 
using  the  Microsoft  Surface  Pro  and  Audacity  audio  recording  and  editing 
software.  The  tablet  platform  was  used  for  the  demo  recording  because  it  is 
similar  to  the  platform  available  to  the  IP.  Audacity  is  free,  open  source  software 
and  works  on  most  common  computer  operating  systems,  and  we  used  it  to 
modify  and  manipulate  voice  recordings.  This  allows  IP  to  simulate  as  if  several 
different  people  (voices)  were  used  in  production,  thus  enhancing  the  realism  of 
the  CAS  scenario  where  typically  multiple  agencies  would  communicate  with  PUI 
during  the  mission. 

Once  each  radio  communication  is  recorded,  the  individual  recorded  audio 
files  are  inserted  into  the  scenario.  Each  audio  file  is  associated  with  specific 
location  along  the  route  of  flight  for  the  simulated  aircraft.  The  audio  files  are 
linked  to  this  specific  location,  and  when  JSON  file  is  imported  Unity  creates  an 
event  that  triggers  the  radio  call.  When  the  aircraft  reaches  a  predetermined 
location  on  the  flight  route  it  activates  the  trigger  and  plays  the  audio.  The  PUI  is 
expected  to  remember  or  record  the  information  passed  in  the  radio 
communication,  and  update  his  (her)  SA  accordingly. 

Ambient  helicopter  rotor  sound  was  also  added  for  realism,  and  an  effort 
was  made  not  to  create  interference  with  the  radio  communications.  Our  intent 
was  that  PUI  would  easily  understand  the  audio,  process  all  information  offered 
through  the  interface  to  enhance  his  (her)  SA,  and  be  able  to  provide  correct 
answers  when  tested  through  the  interface. 
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F.  CHAPTER  SUMMARY 

In  this  chapter  we  detailed  the  creation  of  the  IP  CAS  scenario  builder 
interface  and  the  PUI  CAS  scenario  execution  interface,  and  explained  the 
capabilities  that  each  interface. 
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VI.  FEASIBILITY  TESTING  AND  USER  STUDY 


A.  INTRODUCTION 

This  chapter  reviews  the  feasibility  of  the  part  task  trainer  as  a  training  aid 
and  comments  on  the  design  of  the  future  usability  study.  The  goal  of  feasibility 
study  was  to  provide  an  initial  assessment  of  the  CAS  PTT  and  its  technical 
capabilities.  The  aim  was  to  determine  if  the  current  hardware  and  software 
configuration  could  offer  capabilities  needed  for  training  CAS  operations  by  a 
pilot.  A  formal  user  study  was  designed  to  acquire  valuable  feedback  on  the 
usability  of  the  PTT’s  interfaces  and  to  gather  information  for  its  future 
improvements. 

B.  FEASIBILITY  TESTING 

1.  Visuals 

The  trainer  provides  an  instantaneous  sixty-degree  field  of  view  of  the 
battlespace  virtual  environment.  This  is  less  than  the  normal  field  of  view  (FOV) 
for  full  aircraft  simulators,  but  allowing  a  user  to  rotate  the  camera  view  and 
observe  a  much  larger  portion  of  battlespace  compensates  for  this.  This  is  similar 
to  the  Army's  Dismounted  Soldier  Training  System  that  provides  an 
instantaneous  sixty  degree  FOV  augmented  by  the  ability  to  move  the  FOV 
direction  to  increase  the  virtual  environment  area  observed  (Bink  et  al.  2015,  3). 

The  terrain  graphics  and  other  3D  models  render  completely  throughout 
the  simulation.  The  graphics  refresh  at  a  sufficient  rate  throughout  the  simulation: 
the  lowest  rate  observed  was  76  frames  per  second  (FPS)  when  the  moving  map 
segment  of  PUI  interface  was  open.  The  average  refresh  was  between  the  mid- 
80  to  mid-90  FPS  throughout  the  simulation.  All  of  these  are  above  the  industry 
accepted  minimum  of  60  FPS. 
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2. 


Audio 


Each  radio  call  is  audible  and  clear.  Audacity's  sound  editing  function 
provides  an  appropriate  amount  of  static  noise  to  sound  like  an  actual  radio 
transmission.  The  static  noise  does  not  interfere  with  the  clarity  of  the  radio 
dictation,  which  is  desired  effect.  The  sound  editing  also  makes  each  agency 
sound  like  different  voices  even  though  the  same  person  recorded  most  of  the 
radio  calls. 

3.  User  Interface 

The  user  interface  responded  without  error  to  all  mouse  inputs.  The  touch 
screen  responded  well  with  the  only  issue  on  the  CDU  interface.  Occasionally, 
the  initial  touch  did  not  register  and  the  user  was  required  to  touch  the  screen  a 
second  time  to  activate  the  answer  touch  button.  Multiple  tests  showed  that 
button  responded  on  the  second  touch  each  time.  Increasing  the  CDU  button 
size  reduced  the  number  of  second  touches  required,  but  it  did  not  eliminate 
them  entirely.  The  first  time  the  CDU  pops  up  on  the  screen  it  still  requires  a 
second  touch  to  register.  The  touch  screen  works  on  the  first  touch  all  other 
times.  We  believe  this  issue  to  be  a  bug  in  our  Unity  code. 

C.  DESIGN  OF  USABILITY  STUDY 

The  intent  of  the  user  study  is  to  determine  usability  of  IP  and  PUI 
interfaces;  this  includes  efficiency,  effectiveness  and  user  satisfaction  with  those 
interfaces.  Our  desire  was  to  find  out  how  well  the  tablet  PC  and  this  PTT 
performed  and  gather  users’  opinions  about  different  elements  of  both  interfaces. 

Subjects  for  the  study  will  be  selected  from  the  NPS  student  population;  a 
condition  for  each  person’s  participation  is  their  previous  experience  with  aviation 
training  systems  and  close  air  support  operations.  Experienced  CAS  users  will 
likely  find  any  error  or  omission  in  the  representation  of  the  CAS  scenario  and 
provide  feedback  on  the  changes  required  to  fix  those  issues.  The  study  will  also 
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collect  information  about  subjects’  aviation  and  CAS  experience.  Appendix  A 
details  the  elements  of  all  questionnaires,  including  demographics  data. 

The  usability  study  shall  consist  of  a  familiarization  with  the  interface, 
followed  by  the  session  during  which  subjects  will  use  the  scenario  builder  and 
then  execute  the  scenario  in  the  role  of  the  PUI.  Subjects  will  provide  their 
feedback  on  the  usefulness  and  feasibility  of  the  system  using  a  questionnaire 
after  each  phase  (builder  phase  and  execution  phases).  Appendices  B  and  C 
detail  elements  of  both  questionnaires. 

All  of  the  study's  sessions  are  to  be  conducted  in  a  computer  lab  at  the 
NPS  Modeling,  Simulation,  and  Virtual  Environments  (MOVES)  Institute.  Only  the 
study  administrator  (a  researcher)  and  the  subject  will  be  present  in  the  room. 
Each  session  is  designed  to  be  video  recorded  to  look  for  behavioral  responses  if 
needed  (example:  user’s  posture,  use  of  tablet  platform  and  input  modalities 
during  each  session).  The  Microsoft  Surface  Pro  4  with  the  touch  screen, 
keyboard  and  a  detached  mouse  will  be  used  as  the  PTT  platform  for  each 
subject. 

Each  subject's  study  session  will  begin  by  viewing  prerecorded 
instructional  video  on  how  to  use  the  CAS  PTT  scenario  builder  interface  (IP 
interface)  and  the  execution  interface  (PUI  interface).  The  training  videos  were 
created  to  ensure  a  complete  and  consistent  level  of  familiarization  for  each 
subject.  After  watching  the  training  videos  the  subject  will  be  asked  to  build  a 
complete  CAS  scenario  using  the  IP  interface.  Each  subject  built  scenario  will  be 
from  the  same  template.  The  template  will  contain  all  ACM  and  entity  locations, 
the  route  checkpoints,  and  the  9  Line  information  to  ensure  a  uniform  experience 
for  all  subjects  (the  template  will  be  presented  to  each  subject  on  a  piece  of 
paper.)  Each  subject  shall  be  given  the  opportunity  to  utilize  both  the  mouse  and 
the  touch  screen  for  input  in  building  the  scenario. 

At  the  completion  of  building  the  scenario,  the  subjects  will  begin  the 
execution  phase  of  the  training.  Each  subject  will  be  given  a  sheet  of  paper  and 
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pen  to  record  in-flight  information.  The  sheet  of  paper  simulates  the  pilot's 
kneeboard  in  the  cockpit;  they  will  be  allowed  to  record  any  information  they 
deem  necessary.  The  scenario  is  designed  to  give  the  subjects  a  chance  to 
become  familiar  with  the  interface  and  use  SA  tools  before  they  begin  to  receive 
SA  information  and  answer  the  questions.  During  the  scenario  subjects  will  be 
asked  to  use  both  the  mouse  and  the  touch  screen  to  evaluate  usability  of  each 
input  method.  The  PTT  training  session  will  be  completed  by  listening  to  all  of  the 
radio  communications  and  answering  the  SA  questions.  PTT  will  keep 
information  about  subjects’  performance,  and  they  will  not  be  provided  with  their 
score  during  the  session. 

Subjects  will  be  able  to  ask  any  question  or  make  verbal  comments  during 
and  after  the  session.  Upon  the  completion  of  the  study,  results  of  subject’s 
performance  will  be  reviewed  with  each  person.  All  collected  data  will  be  used  to 
determine  future  improvements  to  the  CAS  PTT. 
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VII.  CONCLUSIONS  AND  FUTURE  WORK 


A.  CONCLUSIONS 

During  this  study,  we  were  able  to  design  and  build  a  tablet-based  CAS 
part  task  trainer  that  focuses  on  training  SA.  The  tablet  PC  platform  using  an 
open  software  game  engine  and  commercially  available  3D  models  were  both 
low  cost  easy  to  use.  All  essential  elements  of  the  CAS  environment  were 
replicated  and  presented  capably  by  the  system.  Based  on  this,  the  tablet  PC 
running  game  engine  software  proved  to  be  a  feasible  training  option. 

Using  the  tablet  PC  to  create  a  compelling  CAS  environment  and  user 
experience  was  successful  too.  The  tablet  provided  both  visual  and  audio 
stimulus  to  the  user.  The  user  received  required  information  via  these  two  modes 
and  was  able  to  answer  SA  questions.  The  remodeled  user  interface  allows  a 
scenario  to  encompass  the  entire  CAS  mission,  expanding  the  capability  of  the 
original  system  originally  developed  by  Attig  (2016).  Also,  the  new  scenario 
generating  interface  introduced  a  virtual  map  tool,  and  provided  a  familiar  way  for 
pilots  to  create  scenarios.  Expedient  graphic  based  generation  of  numeric  data 
like  coordinates  provided  a  much  quicker,  intuitive  way  of  setting  up  the 
scenarios,  including  a  visual  feedback  and  confirmation  to  the  user. 

The  system  does  have  its  limitations  though.  The  difficulty  predominantly 
lies  in  time  required  to  create  the  radio  calls  and  insert  the  questions  at  the 
correct  location.  This  would  need  to  be  remedied  before  the  system  can  meet  its 
true  potential.  Pilot  interaction  via  PUI  interface  is  limited  to  answering  the 
questions  and  repositioning  the  icons  on  the  screen.  We  believe  that  additional 
decision-making  system  capabilities  and  user  interactions  would  greatly  enhance 
user  experience.  While  the  questions  asked  via  PUI  interface  do  an  adequate  job 
of  evaluating  the  PUI's  SA  on  spatial  elements  and  time  driven  events,  further 
research  must  be  conducted  to  determine  if  the  CAS  PTT  does  in  fact  improve 
PUI's  SA.  The  results  of  our  user  study,  once  acquired,  would  provide  advices  for 
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addition  improvements  to  the  CAS  PTT,  and  create  better  virtual  training 
environment. 

The  trainer  can  undoubtedly  serve  as  augmentation  to  current  training 
solutions  available  in  this  domain  -  it  is  well  past  the  time  that  pilot  ground 
training  advances  beyond  printed  publications.  This  PTT  is  a  promising  direction 
in  using  commercial  off-the-shelf  solutions  to  implement  the  low  cost  VR  training 
system  for  the  benefit  of  Marine  Corps  aviation.  The  low  cost,  easy-to-use  and 
easy-to-field  system  like  this  should  make  sure  that  it  ends  up  in  the  hands  of  all 
pilots  throughout  the  fleet. 

B.  FUTURE  WORK 

The  following  improvements  and  additions  to  the  system  could  be 
suggested: 

1.  Improved  Radio  Communications  and  Question  Entry 

Improving  the  radio  call  input  system  would  greatly  increase  the  speed  of 
creating  scenarios.  Currently,  radio  calls  are  recorded  individually  and  then 
placed  at  the  appropriate  location  in  the  scenario.  This  is  time  consuming  and  it 
requires  a  new  set  of  radio  calls  for  each  scenario.  Many  CAS  radio  calls  are 
very  formulaic  and  could  be  easily  reused.  The  shell  of  common  radio  calls  minus 
the  9  Line  inputs  could  be  preloaded  into  the  system.  Inputs  into  the  objective 
area  and  9  Line  builder  could  then  generate  specifics  like  heading,  BP,  and 
artillery  gun  target  line  using  a  text-to-speech  capability. 

Increasing  the  speed  of  entering  the  question  into  the  interface  is  another 
addition  that  would  speed  the  creation  of  scenarios.  Currently  all  questions  must 
be  manually  typed  into  the  interface  or  JSON  file.  Uploading  a  file  of  premade 
questions  from  a  CSV  file  or  having  a  list  of  questions  to  choose  from  via  the 
interface  would  reduce  time  required  to  build  the  scenario. 

Once  these  new  radio  communication  and  question  entry  formats  were 
completed,  it  would  be  beneficial  to  have  a  new  method  of  placing  the  event 
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location  in  the  scenario.  The  exact  location  the  event  is  determined  now,  but  after 
placing  the  radio  or  question  event,  the  scenario  must  be  ran  to  ensure  that  event 
does  not  overlap  with  other  events.  It  would  be  very  helpful  to  have  an 
appropriate  graphical  or  time  display  interface  that  shows  the  distance  along  a 
route  where  the  radio  communication  is  completed  or  where  a  question  is  asked. 

2.  Increase  User  Interaction  during  the  Execution  of  CAS 
Scenario 

The  PTT  only  requires  the  user  to  answers  questions  and  reposition  icons 
when  SA  is  evaluated.  To  take  the  PTT  capability  to  the  next  level,  the  users 
should  make  decisions  during  the  training  and  be  tested  how  well  they  able  to 
project  future  events.  Some  inputs  to  meet  this  end  would  be  to  ask  a  user  to 
determine  flight  path  based  on  events  in  the  scenario.  Additionally,  the  user  could 
be  asked  to  make  radio  calls.  The  former  would  require  an  additional  input,  but 
could  be  done  using  either  the  mouse  or  touch  screen.  The  latter  could  be  done 
with  the  existing  microphone  on  computer  platform  and  speech-to-text  software. 

3.  Usability  Study 

Getting  the  input  of  IP  SMEs  on  the  usability  of  the  PTT  will  provide  some 
immediate  feedback  on  the  PTT's  usefulness.  The  appropriateness  and  quality  of 
the  IP  and  PUI  interfaces  and  the  PUI  immersive  environment  and  scenario  are 
particular  areas  that  could  see  either  improvement  or  validation  with  SME 
feedback.  Ultimately,  the  SMEs  would  provide  a  good  barometer  on  if  the  PTT  is 
a  step  in  the  correct  direction  and  provide  inputs  to  improve  immediate 
shortcomings  in  the  trainer. 

4.  Transfer  of  Training  Study 

A  transfer  of  training  study  provides  insight  on  the  effectiveness  of  the 
CAS  PTT  to  train  SA  techniques.  Designing  a  study  to  evaluate  how  PUIs 
performed  during  CAS  training  missions  in  the  flight  simulator  or  aircraft  would 
verify  if  this  CAS  PTT  is  improving  PUI's  situation  awareness.  This  would  also 
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provide  valuable  understanding  about  both  the  PTT's  training  scenarios  and  the 
PTT  medium  used  to  deliver  the  training.  IP's  could  evaluate  how  well  users  of 
the  PTT  understood  the  CAS  environment  and  then  made  well-informed 
decisions  about  PUIs  skill  remediation  and  future  training. 

5.  Expanded  Mission  Capability 

There  are  opportunities  to  expand  mission  capability  of  the  CAS  PTT.  One 
of  them  would  be  to  create  more  complex  CAS  mission  sets  by  adding  additional 
entities  and  environmental  factors  for  the  PUI  to  interact  with.  The  addition  of  a 
second  aircraft  in  the  PUI's  flight  element  would  increase  the  realism  of  the  CAS 
situation  and  require  the  PUI  to  maintain  SA  on  a  virtual  wingman.  Adding  fixed 
wing  or  other  rotary  wing  CAS  aircraft  to  the  scenario  would  also  increase  the 
complexity  and  realism  because  there  are  frequently  more  than  one  CAS  flight 
element  in  an  objective. 

Also,  due  to  the  flexible  nature  of  the  part  task  trainer  other  mission  sets 
could  be  added.  The  Forward  Air  Controller  (Airborne)  (FAC(A))  mission  is 
conducted  to  support  CAS  missions  and  would  be  a  natural  addition  to  the  PTT. 
Once  the  CAS  mission  complexity  is  increased  it  should  be  reasonable  to  add 
FAC(A)  training  capability.  Assault  Support  missions  like  Combat  Assault 
Transport  would  be  another  possible  addition.  Combat  Assault  Support 
requirements  are  very  similar  to  CAS,  as  they  require  precise  timing,  movement, 
and  fires. 

6.  Integration  with  other  Training  Systems 

Currently,  pilots  use  the  Joint  Mission  Planning  System  (JMPS)  to  plan 
flight  missions.  JMPS  is  a  laptop  computer  based  system  that  contains  extensive 
map  and  terrain  data  files.  The  program's  most  used  functions  are  for  flight 
routing,  fuel  usage,  and  import  of  ACMs  into  aircraft  computer  systems.  A  JMPS 
to  CAS  PTT  interface  could  possibly  use  the  map  and  terrain  data  of  JMPS  to 
generate  the  3D  terrain  in  Unity  or  incorporate  the  JMPS  ACM  and  threat  data 
into  CAS  scenario. 
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APPENDIX  A.  DEMOGRAPHIC  QUESTIONNAIRE 


Thank  you  for  participating  in  this  study.  Please  fill  in  the  following  questionnaire. 

1.  What  is  your  age?  _ 

2.  What  is  your  sex?  _ 

3.  Current  rank?  _ 

4.  How  many  years  do  you  have  on  active  duty?  _ 

5.  Your  primary  MOS/Designation:  _ 

6.  How  many  years  of  experience  flying  military  aircraft? _ 

7.  What  types  of  military  aircraft  have  you  flown? 

a.  _ 

b.  _ 

c.  _ 

8.  Number  of  hours  flying  military  aircraft:  _ 

9.  Was  Close  Air  Support  (CAS)  a  mission  set  for  your  aircraft?  Check  one 
answer: 

O  NO  -  go  to  question  10. 

O  YES  -  answer  the  following  questions: 

If  'Yes': 

A:  How  many  years  of  CAS  experience  do  you  have? _ 

B:  How  many  CAS  mission  sorties  (training  and  actual)  would  you 
estimate  you  have  flown?  Check  one  answer: 

Less  than  10  10  -50  50  -  100  100  -200  Over  200 

□  □  □  □  □ 

C:  Were  you  ever  an  instructor  pilot  for  CAS  training?  Check  one 
answer: 

□  Yes  DNo 
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D:  What  would  rate  your  CAS  proficiency  as? 

Novice  Advanced  Competent  Proficient 
Beginner 

□  □  □  □ 


Expert 

□ 


10.  Do  you  play  games? 


O  NO  -  go  to  question  1 1. 

O  YES  -  answer  the  following  questions: 


Type  of  Game 

(check  all  that 
apply) 


Devices 

(Check  all  answers  that  apply). 


How  often? 

(Check  one  answer  and  enter  your  usage 
hours.) 


O  First  Person 
Shooter 


I  play  them  on  (check  all  that  apply): 

O  Computer  O  Game  Console 

O  Tablet.  Ipad  O  Smartphone 


O  Daily 

Enter  # 
of  hours: 


O  Weekly  O  Monthly  O  Rarely 


Enter  #  of 
hours: 


Enter  #  of 
hours: 


O  Online 
multiplayer 
games 


play  them  on  (check  aN  that  apply): 

O  Computer  O  Game  Console 

O  Tablet.  Ipad  O  Smartphone 


O  Daily 

Enter  # 
of  hours: 


O  Weekly  O  Monthly  O  Rarely 


Enter  #  of 
hours: 


Enter  #  of 
hours: 


O  Adventure, 

Fantasy, 

Role  Playing 
Games 


I  play  them  on  (check  all  that  apply): 

O  Computer  O  Game  Console 

O  Tablet,  Ipad  O  Smartphone 


O  Daily 

Enter  # 
of  hours: 


O  Weekly 

Enter  #  of 
hours: 


O  Monthly  O  Rarely 

Enter  #  of 
hours: 


O  Other  games. 

Enter  the  genre: 


I  play  them  on  (check  all  that  apply): 

O  Computer  O  Game  Console 

O  Tablet.  Ipad  O  Smartphone 


O  Daily 

Enter  # 
of  hours: 


O  Weekly  O  Monthly  O  Rarely 


Enter  #  of 
hours: 


Enter  #  of 
hours: 


54 


11.  Were  you  required  to  use  training  simulations  or  simulators  at  any  point  in 
your  career?  Eg  -Flight  Sims,  Part  Task  Trainers,  MTWS,  VBS,  ISMT,  DVTE  suite 

□  Yes  DNo 

O  NO  -  you  are  complete  with  answering  questions. 

O  YES  -  answer  the  following  questions: 

A.  Enter  the  names  of  those  simulations,  what  skills  were  they  used  to 
train,  how  many  hours  of  training  in  total,  and  the  date  of  last  usage? 

Note***  If  you  do  not  remember  the  name  of  the  simulation,  then  please  enter 
its  closest  description  instead. 

1.  Simulation  #1 : _ 

Skills: _ 

Total  number  of  hours  (approximate): _ 

Date  of  last  use  (approximate):  _ 

2.  Simulation  #2: _ 

Skills: _ 

Total  number  of  hours  (approximate): _ 

Date  of  last  use  (approximate):  _ 

3.  Simulation  #3: _ 

Skills: _ 

Total  number  of  hours  (approximate): _ 

Date  of  last  use  (approximate):  _ 

4.  Simulation  #4: _ 

Skills: _ 

Total  number  of  hours  (approximate): _ 

Date  of  last  use  (approximate):  _ 
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APPENDIX  B.  IP  QUESTIONNAIRE 


Thank  you  for  participating  in  this  study.  Please  fill  in  the  following 
questionnaire. 

Think  about  session  that  you  just  completed,  and  reply  to  following  questions 
regarding  your  experience  in  building  the  CAS  scenario: 


1 .  What  was  the  quality  of  each  visual  element  of  the  interface  you  have  just 
experienced?  Check  one  answer  for  each  characteristic: 


Legibility  of 

the  text  font: 

1 

2 

3 

4 

5 

completely 

somewhat 

average 

somewhat 

completely 

illegible 

illegible 

legible 

legible 

□ 

□ 

□ 

□ 

□ 

Legibility  of 

the  text  font: 

1 

2 

3 

4 

5 

completely 

somewhat 

average 

somewhat 

completely 

illegible 

illegible 

legible 

legible 

□ 

□ 

□ 

□ 

□ 

Contrast  (text/foreground  and  background): 

1 

2 

3 

4 

5 

completely 

somewhat 

average 

somewhat 

completely 

illegible 

illegible 

legible 

legible 

□ 

□ 

□ 

□ 

□ 

2.  What  can  you  say  about  the  amount  of  information  and  guidance  provided  to 
you  via  that  interface  to  help  you  design  CAS  scenario?  : 


1 

2 

3 

4 

5 

completely 

somewhat 

sufficient 

more  than 

too  much 

insufficient 

insufficient 

sufficient 

□ 

□ 

□ 

□ 

□ 

3.  Suggest  how  to  improve  that  guidance: 
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4.  What  can  you  say  about  the  clarity  of  instructions?  : 


1 

2 

3 

4 

5 

very 

somewhat 

average 

somewhat 

very 

difficult 

difficult 

easy 

easy 

□ 

□ 

□ 

□ 

□ 

5.  Suggest  how  to  improve  clarity  of  instructions: 


6.  How  was  the  ease  of  entering  information  scenario?  : 

1  2  3  4  5 

very  somewhat  average  somewhat  very 

difficult  difficult  easy  easy 

□  □  □  □  □ 

7.  Suggest  how  to  improve  entering  information  scenario: 


8.  How  useful  was  it  to  have  2D  Map  preview  display  during  scenario  building? 


1  2 
completely  somewhat 
useless  useless 

□  □ 


3 

4 

5 

average 

somewhat 

completely 

useful 

useful 

□ 

□ 

□ 

9.  How  was  the  level  of  user  input  into  the  interface  to  generate  a  useful  CAS 


scenario?  : 

1 

2 

3 

4 

5 

completely 

somewhat 

sufficient 

more  than 

too  much 

insufficient 

□ 

insufficient 

□ 

□ 

sufficient 

□ 

□ 
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10.  Was  there  any  additional  suggestion  on  how  to  improve  interface  that  you 
just  tested? 

A.  _ 

B.  _ 

C.  _ 
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APPENDIX  C.  PUI  QUESTIONNAIRE 


Thank  you  for  participating  in  this  study.  Please  fill  in  the  following 
questionnaire. 


Think  about  session  that  you  just  completed,  and  reply  to  following  questions 
regarding  the  training  scenario: 


1.  How  was  the  representation  of  helicopter  movement  through  the  battle  space? 

Check  one  answer: 


1 

2 

3 

4 

5 

completely 

somewhat 

average 

somewhat 

completely 

inaccurate 

inaccurate 

accurate 

accurate 

□ 

□ 

□ 

□ 

□ 

2.  Between  the  radio  communications  and  2D  map  chip,  how  was  the  quantity  of 
information  that  you  were  provided  with  to  understand  the  situation, 
particularly  the  battle  space  and  the  CAS  mission  details?  Check  one  answer: 


1 

2 

3 

4 

5 

completely 

somewhat 

average 

somewhat 

completely 

insufficient 

insufficient 

sufficient 

sufficient 

□ 

□ 

□ 

□ 

□ 

How  was  the  2D  Map  chip?  Check 

one  answer: 

1 

2 

3 

4 

5 

not  helpful 

somewhat 

average 

somewhat 

very  helpful 

at  all 

not  helpful 

helpful 

□ 

□ 

□ 

□ 

□ 

4.  Please  evaluate  the  appearance  of  Comm/Nav/Data  display  that  was  used  to 
receive  info  and  to  answer  the  questions  (shown  in  lower  left  corner  of  your 
display  during  the  session).  Check  one  answer  for  each  characteristic: 


Legibility  of  the  text  font: 

1 

2 

3 

4 

5 

completely 

somewhat 

average 

somewhat 

completely 

illegible 

illegible 

legible 

legible 

□ 

□ 

□ 

□ 

□ 
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Legibility  of  the  text  font: 

1 

2 

3 

4 

5 

completely 

somewhat 

average 

somewhat 

completely 

illegible 

illegible 

legible 

legible 

□ 

□ 

□ 

□ 

□ 

Contrast  (text/foreground 

and  background): 

1 

2 

3 

4 

5 

completely 

somewhat 

average 

somewhat 

completely 

illegible 

illegible 

legible 

legible 

□ 

□ 

□ 

□ 

□ 

Ease  of  answering  questions  using  mouse  click: 

1 

2 

3 

4 

5 

very 

somewhat 

average 

somewhat 

very 

difficult 

difficult 

easy 

easy 

□ 

□ 

□ 

□ 

□ 

Ease  of  answering  questions  by  tapping  on  a  touchscreen: 

1 

2 

3 

4 

5 

very 

somewhat 

average 

somewhat 

very 

difficult 

difficult 

easy 

easy 

□ 

□ 

□ 

□ 

□ 

5.  What  method  did  you  prefer  to  enter  your  answers:  using  a  mouse  (mouse 
click)  or  tap  on  a  touchscreen?  Check  one  answer  only. 

Mouse  click  Tap  on  the  touchscreen 

□  □ 


6.  Evaluate  the  radio  communications.  Check  one  answer  for  each  characteristic: 

Radio  calls  were  presented  at  the  appropriate  time  for  the  mission: 


1 

2 

3 

4 

5 

never 

rarely 

neutral 

mostly 

always 

□ 

□ 

□ 

□ 

□ 

Radio  communications  were  loud  and  clear: 

1 

2 

3 

4 

5 

never 

rarely 

neutral 

mostly 

always 

□ 

□ 

□ 

□ 

□ 
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The  scenario  radio  calls  contained  all  required  CAS  radio  communications: 

1  2  3  4  5 

never  rarely  neutral  mostly  always 


□ 


□ 


□ 


□ 


□ 


7.  Evaluate  the  camera  view  rotation  capability.  Check  one  answer  for  each 
characteristic: 

The  ability  to  rotate  or  move  the  camera  view  was  beneficial  for  the  mission: 


1 

not  helpful 
at  all 

□ 


somewhat 
not  helpful 

□ 


3 

average 

□ 


somewhat 

helpful 

□ 


very  helpful 

□ 


How  often  did  you  use  the  camera  view  rotation  feature: 

12  3  4 

never  seldom  occasionally  frequently 

□  □  □  □ 


8.  Evaluate  the  compass  heading  indicator.  Check  one  answer  for  each 
characteristic: 

The  compass  was  helpful  for  orientation  during  the  training  session: 

Yes  No 

□  □ 

How  many  times  did  you  reference  the  compass  to  know  aircraft  heading: 


1  2 

never  once 

□  □ 


3  4  5 

2  -  3  times  4-5times  >  5  times 

□  □  □ 


9.  Do  you  have  a  preference  for  the  type  of  camera  view  the  trainer  uses?  Check 
one  answer: 


1st  Person 

□ 


3rd  Person 

□ 


No  Preference 

□ 
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10.  Would  you  recommend  this  type  of  trainer  to  be  used  to  teach  CAS 

procedures  and  techniques  to  enhance  situational  awareness?  Check  one 
answer. 


Yes  No 

□  □ 

11.  What  are  your  top  3  suggestions  for  changes  or  improvements  to  the  trainer? 

1.  _ 


2. 


3. 
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